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A new method for efficient study of Kanji 
using mnemonics and software 



Chris Houser*, Shigeki Yokoi** and Takami Yasuda** 

*Kinjo Gakuin University, Omori 2-1723 Moriyama-ku Nagoya 463-8521 Japan, 

houser@kinjo-uMc.jp 

**Nagoya University, Furo-cho Chikusa-ku Nagoya 464-8601 Japan 

Japanese children spend hundreds of hours, over nine years, studying some 2000 
written characters called kanji. Incredibly, most foreign adults attempt to study 
the kanji using the same method. But without these hundreds of hours, their 
efforts generally fail. In Remembering the Kanji, James Heisig presents a radical 
method for studying kanji. In only 200 hours, Heisig claims, adults can learn the 
kanji. A wonderful improvement! But few students follow his method; most 
complain that 200 hours is still too long. This paper introduces a refinement of 
Heisig 's technique, a refinement combining modem memory theory with 
software, a refinement reducing the required time to 40 hours. The first author, a 
forgetful kanji neophyte, learned the kanji with this method, studying an hour a 
day, five days a week, for two months. His recall exceeds 95%, approximating 
native Japanese. This paper targets both teachers and students of Japanese as a 
foreign language, providing the knowledge and software required to rapidly 
learn the kanji , and inviting them to participate in a wider experiment using 
these new technologies. 

Keywords. CALL, Kanji, SuperMemo, Efficient study 



1 The Kanji 

Perhaps the most difficult part of learning Japanese is memorizing its enormous character set: the 2000-odd 
kanji. These characters were imported from China into Japan. Because each character was imported several 
times over the centuries, while the Chinese and Japanese languages were evolving, each character now has 
multiple readings and meanings. As a result, the Japanese writing system is arguably the world's most 
complex. 

Japanese children study these kanji for hundreds of hours over nine years of schooling. They start studying 
when six years old, before they have developed the ability to abstract, and hence can learn the characters 
only by muscle memory: They write the characters repeatedly, typically 20 times each. This method works, 
but imperfectly: Even after all this study, and the review that comes with daily use, adult Japanese forget 
some characters. 

Most foreigners studying Japanese as a foreign language (JFL) try to learn the kanji using the same method: 
They write the characters repeatedly, perhaps while verbalizing the character's meanings and readings [7]. 
But since few adult JFL students have the hundreds of hours this method requires, most fail [4], 



2 Heisig ’s method for studying the kanji 

Heisig [4] offers an alternative study method. 

Goal. Heisig's method allows adult JFL students to learn the writing and a single meaning of 2042 kanji. 
This is a narrow goal: Students concentrate on learning this writing and single meaning, and postpone 
learning other meanings, all readings, and the multiple character compounds. 

Method. Since Heisig targets adults, he is able to use a sophisticated method, a method beyond the grasp of 
six year olds. He is able to use a rational method for learning kanji. Heisig prepared his method by 
1 . assigning each character a keyword (its single meaning), 



2. splitting each character into a handful of parts, 

3. ordering the characters so that parts precede their uses, and 

4. inventing a mnemonic story to help recall each character’s parts. 



The keyword is usually the most common of the several Japanese meanings. The parts come from various 
sources: Some are simpler kanji ; others are primitives - collections of commonly occurring strokes. Some of 
these primitives were identified centuries ago by Chinese and Japanese linguists (who call them “radicals”); 
other primitives were simply invented by Heisig. In all, Heisig uses a few hundred parts. The crux of his 
method: 

Each character is learned, not as a mass of random strokes, but as a logical collection of parts. 

For example, consider the kanji with the keyword revise. This character has nine meaningless strokes, 
which prove quite a challenge to remember. But this same character has only two parts with the keywords 
words and nail - meaningful words which are much easier to remember. In effect, Heisig splits this 
character into these two parts, making a kind of equation: revise = words + nail. Most non-Japanese find 
this equation much simpler to recall than a meaningless jumble of nine strokes. When Heisig’s students 
come to study revise, they have already learned the two parts - word and nail - since Heisig has sorted the 
kanji so that these parts precede their use in revise. By combining two previously learned parts, students 
easily remember this new character. But Heisig makes remembering even easier by providing a mnemonic 
story: 

REVISE your draft by NAILing down your WORDS. 

The image of “nailing down one’s words” is so strong and logical that after students have read this 
mnemonic once, they will likely remember it for life. 

This contrasts with Japanese students, who practice writing the character repeatedly, and may later 
forget it. 

Heisig’s main contribution is to raise the level of abstraction from strokes to parts. Rather than 
struggling to remember a large, sprawling jumble of meaningless jots and dashes, students effortlessly 
remember a simple story, calling to mind the few parts that compose a kanji: 



words n 
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nail 



Previously memorized parts, 
themselves simpler kanji . . . 




... are combined into a 
new kanii . . . 



REVISE your draft by . . . and memorized via a 

NAJJ.inv down vnur WORDS. simnle. mnemonic storv. 



Study. Heisig has done most of the work: He has assigned the keywords, identified the parts, invented the 
primitives, and sorted the kanji. Students need only read the keyword and story a few times to memorize 
each kanji. Heisig predicts study will require 200 hours - far less than Japanese children spend on rote 
repetition. 

Analysis. Why is Heisig’s method so effective? Here are three explanations. 

Simplicity. The stories are simpler than the kanji y simpler because they have fewer components. Each kanji 
consists of between 1 and 23 strokes; 75% of the kanji have more than seven strokes. But all have fewer than 
seven parts. Now human short-term memory can hold only about seven items [6]. Objects with more than 
seven known components cannot fit in short-term memory, and so cannot be remembered, or even 
recognized. This predicts that students learning strokes will remember 25% of the kanji , but students 
learning parts will remember 100%. 

Abstraction. Practicing strokes engages only muscle memory: Most of the student’s brain remains dormant. 
Heisig’s stories engage the higher faculties of language, actions, settings, events, humor, and metaphor. Such 
meaningful symbolic processing engages more of the brain, and hence is more easily recalled, than mere 
orthographic syntax [8]. Humans recall abstract meanings and stories long after they forget specific 
examples and images [5]. 

Relations. When learned by rote, each kanji , indeed each stroke, must be learned anew: Nothing is 
connected to anything else. When learned by parts, each kanji is connected to previously learned kanji. 
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Heisig’s method is rich in connections. When students learn a character, they are also reviewing its parts. In 
turn, most parts appear repeatedly, and hence are memorized easily. 



As an illustration, consider the daunting 23 strokes of specimen . Stroke-by- stroke memorization is all but 
impossible. But specimen comprises only two parts: gold and oversee . It is easily recalled with a story such 
as GOLD diggers OVERSEE their mineral SPECIMENS. Specimen is studied after gold and oversee have 
been learned, from their own parts, with their own stories. So each step of study is small and simple, but the 
steps build on each other - primitives are woven into kanji , which are in turn used to build further kanji - 
until a vast web of rich connections is built up in the student’s mind. 
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or primitives (marked with *) 

review flash cards, need a scheduling system to study, review, and test. 



Problem: Still too difficult! 

Heisig’s method is a great 
improvement over the 
Japanese method, but it is 
not perfect. For Heisig 
provides stories for only the 
first 500 of his kanji, and 
asks readers to invent their 
own stories for the 
remaining 1542. Faced with 
this burden, many of 
Heisig’s students stop 
studying after 500 
characters. And those who 
do continue need unusual 
discipline, need to 
painstakingly construct and 



3 Kanji Can 

Kanji Can (3, 1] is a database with a complete set of 2042 mnemonic stories. The stories are excellent, 
surpassing even Heisig’s first 500: 

• Kanji Can’s stories are shorter, and so easier to recall. . 

• Kanji Can’s stories mention the parts in the order they are written. 

(Compare with Heisig’s story for revise above, which reverses them.) 

Kanji Can embraces Heisig’s method, but extends his materials, and thus solves the problems mentioned 
above. 



4 Flash Cards 



The chief tool of most memorizers is the humble flash card. Flash cards are small paper cards with a 
stimulus written on the front side, and a response on the 
back. When studying foreign language vocabulary, the 
stimulus is typically a word in one’s native language, and the 
response is the word in the foreign vocabulary. When 
studying kanji using Heisig’s method, the stimulus is the 
keyword, and the response is the kanji itself. 




front 



SPECIMENS 

WL omSSS 



back 



Students read the stimulus and try to produce the response. They then check their response against the 
desired response on the back of the card. Cards that were correctly recalled are removed from the deck; 
cards not recalled are shuffled to 

the back of the deck, to be reviewed * rec&IL^^Z?**^ study 

again. Used this way, flash cards ~ ~ 

combine self-testing with review. 

The cards catch mistakes and save 

them, allowing review until the ±time 

student knows them all. Flash cards Repeated study slows 
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are essential when following Heisig ’s method: Studying the stories is so easy that students will doubt they 
are really learning anything until they have been tested! 

Problem: Inefficiency. Using flash cards takes a lot of time. Each card must be hand made. Then each card 
must be tested repeatedly, for only with repetition comes dependable memorization. Memory fades over time, 
but by reviewing partially forgotten material students extend their memories. 

But how frequently should students review? Buzan [2] recommends review after ten minutes, a day, a week, 
a month, and then four months. But are these the best intervals for review? Testing too frequently wastes 
time reviewing material already well known. Testing too infrequently wastes time relearning forgotten 
material. The goal of flash cards is to “catch” learners just before they fall - to remind them just as they are 
about to forget. But the point of forgetting - and hence the optimal review interval - differs for each student, 
and even for each kanji: Some are easier to remember than others. How can we optimize study? 



5 Super Memo 

Super Memo is a general flash card program [9]. Like paper flash cards, these electronic cards can be used to 
review anything, including the kanji. Unlike paper cards, these electronic cards are neat and easily editable, 
but require a PC. Super Memo is better than paper flash cards because it contains a mathematical model of 
human forgetting: It can predict when a student will forget a kanji , and hence compute the best testing time. 
When testing with Super Memo, students tell the program how well they remember each kanji; the program 
uses this information to tune its model to each student, and to each kanji. The result closely approximates 
perfectly timed intervals, and hence maximum efficiency in studying. 

Independent of the nature and amount of material they study, students using Super Memo all learn 
approximately 200 items/minute/year. This means that by studying one minute, every day, for a year, one 
can learn 200 items; or, by studying 10 minutes a day, 2000 items. This is much faster than many other study 
methods; in particular, Super Memo implies results in 1/5 of Heisig ’s time. 

Super Memo’s computerized scheduling provides more than optimal reviews. It also provides an incentive to 
study every day. A student using Super Memo runs the program every day and finds a list of items to review. 
If the student skips a day, the next day she will be confronted with twice as many items! This threat helps 
provide the discipline necessary in learning a large body of material, such as the kanji. (Unfortunately, this 
also means that if the student skips a week, she will be confronted with a mountain of review, and will likely 
quit altogether. Super Memo is not for the timid.) 



6 New technology allows learning the kanji in only 40 hours! 

This paper proposes a new method for learning the kanji, a method combining Heisig ’s novel ideas, Kanji 
Can’s stories, and Super Memo’s reviewing. Heisig provides the tractable goal and the idea of using 
mnemonic stories to recall the writing of kanji in terms of their parts. Kanji Can provides a complete set of 
these mnemonic stories. And Super Memo provides strict scheduling and efficient reviewing and testing. 
The combination of these three educational technologies provides a most efficient kanji learning method: the 
complete set of 2042 kanji can be learned in only 40 hours! 

These 40 hours might be scheduled as 10 minutes a day, every day for a year, or an hour a day, five days a 
week, for two months. Memory manuals claim that an hour’s study a day is optimal: Shorter study sessions 
waste time in frequent physical and mental preparation, longer study sessions induce fatigue, and both 
degrade efficiency [2, 5]. 

The first author learned the kanji in 40 hours by following this method. 

Heisig has greatly accelerated kanji learning for adult JFL students. Kanji Can’s complete set of stories 
enables students to concentrate on studying the kanji. Super Memo provides a well-documented speedup for 
any rote memorization. Combining these three technologies, we can learn the kanji in only 40 hours. 
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A Study on the Relation between 
Touch-typing Skill and Thinking-typing 
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Word processor is more and more widely used as a tool of external ization and 
reflection of thinking in recent years in Japan. In that case, it will be necessary to 
type smoothly words or sentences appearing in the head (hereinafter referred to as 
"thinking-typing"). The experiments were made to study the relation between 
touch-typing skill and thinking-typing. The examinees were 46 non-computer 
majors of the university. The students were asked to type words or sentences 
appearing in their heads on 3 subjects. The touch-typing skill of the students was 
measured by touch-typing exercise software. The results suggested that a 
touch-typing speed of 2 strokes/second is necessary, at least, to type smoothly 
words or sentences appearing in the head. What's more, the results of the 
experiments suggested that learning of touch-typing skill is very effective on the 
increase of thinking-typing speed of the subject that is easy to be thought out. 

Key words: Thinking- typing, Touch-typing, Externalization, Self-evaluation, 

Analysis of variance 



1 Introduction 

The methods for human beings to externalize their thinking are language expression, diagram expression, 
letter expression and so on Among these expressions, letters are widely expressed by word processors in 
recent years in Japan. The method of word processor’s usage has been changed by the popularization of them. 
In other words, the method that uses a word processor to transcribe a manuscript written by handwriting, has 
been changed to the method that uses a word processor in the process of externalization and reflection of 
thinking. With the latter method, it is necessary to type smoothly words or sentences appearing in the head 
(hereinafter referred to as "thinking-typing"). Thinking-typing needs a certain level of typing skill. Although a 
number of studies have been made on typing ^ there are few studies on thinking-typing. 

In the lesson of computer exercise at the university, the first author is raising the level of students’ typing skill 
through touch-typing education and, at the same time, is raising the ability of the students’ utilizing a word 
processor as a tool of externalization and reflection of thinking ^ In the lesson, the experiments of 
thinking-typing by touch-typing were made to study the relation between touch-typing skill and 
thinking-typing. Touch-typing speed and self-evaluation of thinking-typing were adopted as the scale of 
thinking-typing level. The first experiment (Experiment 1) was made in July, 1999, and the second 
experiment (Experiment 2) was made in February, 2000. In this paper, results regarding Experiment 2, and 
comparison between Experiment 1 and Experiment 2 are reported, because results regarding Experiment 1 had 
been reported already ^ 



2 Method 



The experiments of thinking-typing by touch-typing were made in the lesson of the computer exercise for the 
first-year students at the university. In this study, the data of 46 students, whose data of Experiment 1 and 
Experiment 2 were complete, were analyzed. In the experiments, the students typed the following subjects by 
touch-typing. 
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One-way analysis of variance was used to test for significant differences in thinking-typing speed among the 4 
touch-typing levels. As a result, touch-typing skill had main effect in the 3 subjects( 1 S , /jm/ori:7 r =12.46, df 
=3, p <. 01 Imagination: 7^=1 1.31, df= 3, /? < .0 1 Impression: 7 r =23.55, df= 3, p<. 01). What's more, 
Tukey's multiple comparisons test was applied to identify whether there are significant differences in 
thinking-typing speed among the 4 touch-typing levels or not. The results are shown in Table 2. 
Homogeneity subgroup is a group of similar levels whose difference is not significant. In the 3 subjects, there 
were significant differences of thinking-typing speed between the level under 2 minutes and the level over 2 
minutes. These results show that reaching touch-typing level under 2 minutes in Experiment 2 was one of the 
conditions to type smoothly words or sentences appearing in the head. 
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3.2 Relation between Touch-typing Skill and Self-evaluation of Thinking-typing 

Self-evaluation of Experiment 2 was divided into positive self-evaluation and negative self-evaluation to study 
the relation between self-evaluation and touch-typing skill. Positive self-evaluation is "very good", "good" and 
"a little good". Negative self-evaluation is "a little bad", "bad" and "very. bad". As for self-evaluation point, 
positive self-evaluation is 1 point, and negative self-evaluation is 0 point. The mean of self-evaluation point 
of each touch-typing level is shown in Table 3. 



Table 3. Touch-typing skill and self-e valuation 
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One-way analysis of variance was used to test for significant differences in self-evaluation point among the 4 
touch-typing levels. As a result, touch-typing skill had main effect in the typing evaluation of imagination 
and in the typing evaluation of impression (typing evaluation of imagination: F =5 A 1 , df=3,p < .01 typing 
evaluation of impression: ^=4.86, df= 3, /?<.01). What's more, Tukey's multiple comparisons test was 
applied to identity whether there are significant differences about the typing evaluation of imagination and the 
typing evaluation of impression among the 4 touch-typing levels or not. The results are shown in Table 4. 
Typing evaluation of Level of 3 minutes in imagination was significantly lower than other touch-typing levels, 
and typing evaluation of Level of 3 minutes in impression was significantly lower than Level under 1 minute 
and Level of 1 minute. These results show that the students of Level of 3 minutes could not type smoothly 
imagination or impression, comparing with the students of other touch-typing levels. 



[Subjects of Experiment 1 ] 

Subject 1: Type words that you think with shiritori (a Japanese word chain game/ Type them by 
hiragana (Japanese alphabet). The time limit is 3 minutes. 

Subject 2: Type words that you image with "university". Type them by hiragana-kanji (Japanese 
alphabet - Chinese characters) translation. The time limit is 5 minutes. 

Subject 3: Type sentences of your self-introduction. Type them by hiragana-kanji translation. 

The time limit is 1 0 minutes. 

[Subjects of Experiment 2] 

Subject 1: Same as Subject 1 of Experiment 1. 

Subject 2: Type words that you image with "student life". Type them by hiragana-kanji translation. 
The time limit is 5 minutes. 

Subject 3: Type sentences of your impression about the lesson of the computer exercise. 

Type them by hiragana-kanji translation. The time limit is 10 minutes. 

After the experiment, the students evaluated themselves on the 3 subjects. In Experiment 2, the students 
evaluated themselves on whether they could think out words and sentences or not (thinking evaluation), 
whether they could type words and sentences by touch-typing or not (typing evaluation). The evaluation 
standard was divided into 6 levels: "very good", "good", "a little good", "a little bad", "bad" and "very bad". 

The touch-typing skill of the students was measured in the lessons before and after the lesson of the 
experiments. The measurement content is testing typing time of entering Japanese sentences of hiragana 
(about 240 strokes) that were displayed in a monitor at random, by romaji (Japanese Roman characters) input 
and touch-typing. 



3 Results and Discussion 

3.1 Relation between Touch-typing Skill and Thinking-typing Speed 

Touch-typing skill in Experiment 2 was divided into 4 levels: under 1 minute (Level under 1 minute), 

between 1 minute and 2 minutes (Level of 1 minute), between 2 minutes and 3 minutes (Level of 2 minutes), 
between 3 minutes and 4 minutes (Level of 3 minutes). The mean and the standard deviation of 
thinking- typing speed in each touch-typing level are shown in Table 1. Thinking- typing speed in each subject 
was calculated by the next equation. 

L 



s = — 

T Table 1 . Touch-typing skill and thinking-typing speed 



s : Thinking-typing speed in each 
subject 

L : Typing linage in each subject* 




Nurrber of ■ 
persons - 




Thinking-typing speed (linage/minute) 




Touch-typing skill 


Shiritori 
M SD 


Imagination 
M SD 


Expression 
M SD 


T : Time limit in each subject 


Level under 1 minute 


6 


1.13 


0.25 


0.65 


0.16 


1.07 


0.14 


(minute) 


Level of 1 minute 


23 


0.84 


0.23 


0.52 


0.14 


0.72 


0.19 


* Number of letters per line, after 
hiragana-kanji translation, is 40. 


Level of 2 minutes 


12 


0.59 


0.11 


0.35 


0.06 


0.47 


0.11 


Level of 3 minutes 


5 


0.51 


0.13 


0.28 


0.09 


0.42 


0.10 



4 6 0.77 0.27 0.47 0.17 0.67 0.25 



All the examinees 



Table 4. Tukey's multiple comparison of self-evaluation 





Touch-typing skill 


Number of 


Homogeneity subgroup 




persons 


Grl 


Gr2 




Level under 1 minute 


6 


1.00 




Typing evaluation 


Level of 1 minute 


23 


0.96 




of imagination 


Level of 2 minutes 


12 


0.83 






Level of 3 minutes 


5 




0.40 




Level under 1 minute 


6 


1.00 




Typing evaluation 


Level of 1 minute 


23 


0.96 




of impression 


Level of 2 minutes 


12 


0.75 


0.75 




Level of 3 minutes 


5 




0.40 



P -05 / 



3.3 Relation between Learning of Touch-typing Skill and Change of Thinking-typing 
Speed 



The mean of learning ratio of touch-typing skill and the mean of change ratio of thinking -typing speed in each 
touch-typing level of Experiment 2 are shown in Table 5. Learning ratio and change ratio were calculated by 

the next equation. rr71 _ si 

71 fl? — 

a = — r s2 

T 2 



a : Learning ratio of touch-typing skill 

77 : Touch-typing time of Experiment 1 (minute) 

T2 : Touch-typing time of Experiment 2 (minute) 



fi : Change ratio of thinking-typing speed 

si : Thinking-typing speed of Experiment 1 (linage/minute) 

s2 : Thinking-typing speed of Experiment 2 (linage/minute) 



Two-way analysis of variance was used to test for significant differences in the 4 touch-typing levels and the 3 
subjects about change ratio of thinking-typing speed in Table 5. As a result, main effect of the 3 subjects was 
significant F =4.14, df—2, p<. 05). Main effect of the 4 touch-typing levels and interaction were not 
significant. What's more, Tukey's multiple comparisons test was applied to identify whether there are 
significant differences in change ratio of thinking-typing speed among the 3 subjects or not. As a result, there 
were significant differences of change ratio of thinking-typing speed between Subject 3 and other subjects. 
Next, correlation between learning ratio of touch-typing skill and charge ratio of thinking- typing speed is 
shown in Table 6. A moderate significant positive correlation was observed between learning ratio of 
touch-typing skill and change ratio of thinking-typing speed in Subject 3. So it can be said that learning of 
touch-typing skill is very effective on the increase of thinking-typing speed of Subject 3. What is the 
qualitative difference between Subject 3 and other subjects? It is the easiness of thinking. Thinking evaluation 
point in Table 3 expresses the easiness of thinking in each subject. Thinking evaluation point of impression 
(Subject 3) is higher than other subjects. So it is considered that words of impression (Subject 3) was easier to 
be thought out than other subjects. Therefore, it can be concluded that learning of touch-typing skill is very 
effective on the increase of thinking-typing speed of the subject that is easy to be thought out. 



Table 5. Learning ratio of toucfe-typing skill and change ratio of thinking-typing speed 



Touch-typing skill 


Number of 
persons 


Learning ratio 
of touch-typing 


Change ratio of thinking-typing speed 
Subject Subject 2 Subject 3 


Level under 1 minute 


6 


1.69 


1.35 


1.61 


2.03 


Level of 1 minute 


23 


1.58 


1.24 


1.47 


1.93 


Level of 2 minutes 


12 


1.57 


1.18 


1.57 


1.61 


Level of 3 minutes 


5 


1.50 


1.89 


1.47 


1.80 


All the examinees 


46 


1.58 


1.31 


1.51 


1.84 
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Table 6. Correlation between learning ratio of touch-typing 



skill and change ratio of thinking-typing speed 



Change ratio of thinking-typing speed 




Subject 


Subject 2 


Subject 3 


Learning ratio 
of touch-typing 


0.245 


0.089 


0.565** 



**p .01 / 



4 Conclusion 

From what has been discussed about the relation between touch-typing skill and thinking-typing speed, and 
relation between touch-typing skill and self-evaluation of thinking-typing, it can be concluded that a 
touch-typing level under 2 minutes is necessary, at least, to type smoothly words or sentences appearing in the 
head. The speed of 240 strokes in 2 minutes equals 2 strokes/second. 2 strokes are needed to input a 
hiragana. So the speed of 120 hiragana in 2 minutes equals 1 hiragana! stcond. The aim of touch-typing 
education for thinking-typing should be set at 2 strokes/second (1 hiragana! second). What's more, from what 
has been discussed about the relation between learning of touch-typing skill and change of thinking-typing 
speed, it can be concluded that learning of touch-typing skill is very effective on the increase of 
thinking-typing speed of the subject that is easy to be thought out. 
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Several experiments of situated instruction have been done in elementary school. 

We conduct the inaugural experiment on college students. A group of 44 
students who are taking food microbiology course involved in this experiment. 

We designed a science fiction named "Save the Taiwan”, which is a story 
regarding a Microbiology technician handles a disease crisis. A student can learn 
how to deal with the crisis and solve the problem of an infectious disease when 
he uses this CAI software. The evaluation practice consists of four dimensions, 
subject domain cbmands, instructional demand, user interface demand, and 
pragmatic demands. The result of evaluation shows notable effect on college 
students. 

Keywords: situated learning, evaluation of CAI, Microbiology 



1 Introduction 

The advantage of traditional instruction is that the knowledge that students learned can be ” stiff knowledge”. 
The stiff knowledge can not be smoothly applied to solve the actual problem in a real environment (Brown, 
1989). Situated cognition bases on the theory that the learning should be constructed at real situation. Only 
when the learner derive the knowledge from the real situation then he realizes the real meaning of the 
learned knowledge and cherish the value of knowledge and take it as the tool for solving the problem. 
(Cognition and Technology Group at Vanderbilt, 1990) 

There was experiment on elementary school student (Tsu, 1997). The experiment was focus on learning 
simple mathematics calculating. We conduct the experiment on college student, trying to find out if situated 
learning can be succeed on the domain of higher education. The students of Department of Microbiology 
have to take laboratory classes during the period of 4 years college. They always have the problem of how 
and where to apply what they have learned in the class. The instructive goals of laboratory class are diverse. 
Lazarowitz& Tamir (1992) believe that learning is a process of construction. Despite of learning the 
laboratory skill, oral discussion between instructor and students should be part of the learning process. In 
addition, a more inexpensive and more efficient instruction method, such as computer aided instruction or 
simulations, should be adapted in the class. Anchored situated instruction adapts computer technology to 
implement situated cognition theory. (Cognition and Technology Group at Vanderbilt, 1990) hi our software, 
we develop a pseudo but can be real situation, embedded the learning materials in the circumstances. 
Through the interaction between user and CAI software, learner can then practice the process of collecting 
and filtering information as well as the methods to solve the problem. 
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2 Design features of our approach 



The script was written and designed by a professor of Department of Microbiology. We adopt several 
principles in our design. 

1 . Provide vivid circumstance. Multimedia allows us to design a vivid environment, so that the situation can 
avoid over-simplified and lack of context. (Yang, 1995). Multimedia also provides plentiful visual symbols, 
e.g. video images, graphic charts, sound as well as text to make the play more fun and close to real 
environment. 

2. To hide useful information in the story, a learner may have trouble to transfer what he has learned to 
different situation if he was teaching in a simplified or provided obvious cue environment. Whereas, a 
learner can learn to justify what information can be useful and what information is not useful for solving the 
problem in a simulated situation. 

3. To randomize the plot of the story, the learner can memorialize the plot if he has experienced in using this 
CAI software. This can then cause the learning process in vain. We use Random function in our design; one 
of the bacteria was picked randomly and then, in turn, develop different story. 

4. To have productive result, there may be only one answer or solution in a traditional instruction. However, 
in a real environment, there may be more than one solution to solve the problem. They may be pros and cons 
from one to one, but they may all workable. We do not provide firm answer to user instead, we provide an 
open-ended environment for user to construct his own path and solution. 



3 The story 

3.1 Outline of the story 

There is a food mediated infectious disease occurs in a small town that locates at the seashore of Taiwan 
island. Within a few days, this infectious disease has spread to nearby counties and caused many cases of 
death. Tai-shang (see photo 1), the leading actor, a technician of the local public health administration office, 
is responsible for finding out the etiologic agent of the disease. In the story, Tai-shang is facing many 
challenges like the ones in real life. His girl friend, professor and colleagues are all in the plot and interactive 
with him. After the accident happened, he had faced the pressure from his superior, public media, even from 
a local councilor. He must acts like a detective who searches the cue and a scientist who seeks for the truth 
of problem. Finally, with the encouragement of his college instructor, Ta-shang successfully finishes his 
task. 



Photo 1. A clip photo from video " To save Taiwan". 




3.2 Goal of learning 

a.* Cognitive aspect* • 

• a* Assessment of identifying the virus, fatal virus can not be classified from appearance. Thus, the learner 
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has to make a judgement base on the information gathered from the plot and then decide how to 
proceed the process of bacteria identification. 

• b* Learning of the skills of bacteria identification, there are skills, e.g. Stain, biochemical test, can be 

practiced. 

• e* Usage of bacteria identification index table, after the preliminary result of biochemical test, the learner 

needs to learn to use bacteria identification index table for final judgement. 

b.* Attitude • • 

• a* right attitude of science work, through playing the role in the game, the learner can identify the spirit of 

scientific work, diligence and concentration, as the attitude of being a scientist. 

• b* Social caring, since the story has a local background, we hope the learner can improve the caring of 

local society by solving the problem for local society. 



4 Evaluation of our experiment 

There are four criteria of the evaluation of the designated software. The criteria is based on the character the 
teaching subject, human learning theories, and research on user interfaces* Kuittinen, 1998* • The criteria 
consist of four types of requirements: subject domain, instructional, user interface design and pragmatic 
matters. We invited two batches of domain experts, the faculties of Department of Microbiology and 
industry professionals to evaluate subject domain. They focused on examining if the concept and methods of 
this domain are generally applied in our software, which means they checked the relevancy to instructional 
aims. The group of 20 experts showed their positive opinion a the following chart.* Table h *The 
instructional demand is a student-centered approach. We divide a group of 44 college students who are 
taking food microbiology into two groups. The 22 randomly selected members of test team used our 
software for average 6 hours in a period of a week. A cognitive examination was taken after one week. The 
result shows that the test team has better performance in cognitive aspect. Table2* The interactivity, display 
elements and connections between them are examined as the criteria of user interface. We use questionnaire 
to test team and found out that the team members show satisfaction of the user interface.* Table 3* As the 
pragmatic criteria, the hardware and software requirements are evaluated to see if a specific and /or 
expensive equipment or environment is required to use the software. Our software can be used in a common 
Microsoft Windows environment plus Pentium compatible personal computer. A learner can run our 
software either at computer room in campus or at his own PC. 



Table 1 . The result of experts poll 





number 


Min 


Ma 

X 


Averag 

e 


stand 

ard 

deviat 

ion 


varianc 

e 


Can the instructional goal of " assessment of 
identifying pathogenic bacteria " be reached? 


12 


3 


4 


3.67 


0.49 


0.242 


Can the instructional goal of " learning the 
identification methods " be reached? 


12 


2 


5 


4.00 


0.74 


0.545 


Can the instructional goal of " usage of diagnostic 
table for the identification bacteria " be reached? 


12 


2 


5 


4.08 


0.79 


0.629 


Can the instructional goal of " influence on right 
scientific attitude " be reached? 


12 


2 


4 


3.58 


0.79 


0.629 


Can the instructional goal of " influence on social 
caring" be reached? 


12 


2 


4 


3.25 


0.87 


0.750 


Does the content of software cover" common 
foodbome pathogenic bacteria and their 

characteristics"? 


12 


2 


5 


3.92 


0.79 


0.629 


Does the content of software cover" procedures of 
identification of pathogenic bacteria" 


12 


4 


5 


4.33 


0.49 


0.242 


Does the content of software cover" knowledge foi 
assessment of methods used in bacterial 
identification" 


12 


2 


5 


4.17 


0.83 


0.697 
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Table 2. Evaluation on the cognitive improvement of the software 



Item 


n 


Avera 

ge 


SD 


F-value 


df 


t-value 


Score of Midterm 

examination 


Control 

group 


23 


61.96 


6.92 


2.54 


42 


-0.34 


Exp. group 


21 


62.90 


11.03 








Semester score 


Control 

group 


23 


80.09 


9.66 


1.00 


42 


1.59 




Exp. group 


21 


75.48 


9.65 








Cognitive Examination 


Control 

group 


23 


29.74 


6.11 


1.09 


42 


-3.14** 




Exp. group 


21 


41.29 


6.98 








Situated Questions 


Control 

group 


23 


3.57 


2.64 


1.04 


42 


-3.73** 




Exp. group 


21 


6.57 


2.69 








Questions on 
Microbiological Skill 


Control 

group 


23 


26.17 


5.07 


1.08 


42 


-5.47** 


Exp. group 


21 


34.71 


5.28 









Table 3. Evaluation of user interface of the software 





nu 

mb 

er 


Min 


Ma 

X 


average 


standar 

d 

deviati 

on 


variance 


I do not need tutoring before I use this software 


21 


2 


5 


3.95 


0.92 


0.848 


I can easily know how to jump to next screen 


21 


1 


5 


3.76 


1.18 


1.390 


I can exit the software anytime, anywhere. 


21 


2 


5 


4.52 


0.93 


0.862 


I do not have the situation that I can not proceed 
because that I did not memorize the previous 
information while I use this software. 


21 


2 


5 


4.19 


0.87 


0.762 


It's easy to modify my answer before I press the 
"confirm" bottom 


21 


2 


5 


4.10 


0.94 


0.890 


I can receive the system feedback anytime when I use 
the software. 


21 


2 


5 


3.90 


0.77 


0.590 


The system feedback is clear enough and no need to 
be explained. 


21 


2 


5 


3.81 


0.81 


0.662 


I can use the software without reading the user's 
manual in ahead 


21 


2 


5 


4.19 


0.87 


0.762 


I think the execution speed is proper to me. 


21 


2 


5 


3.95 


0.80 


0.648 


I can handle the execution speed of my own. 


21 


2 


5 


3.81 


0.75 


0.562 


I am satisfied the quality of the video. 


21 


1 


5 


3.29 


1.01 


1.014 


I can use the software without knowing how to 
operate Window NT 


20 


3 


5 


4.55 


0.60 


0.366 


Total score 


21 


39 


55 


47.81 


4.12 


16.962 



5 Conclusions 

We completed situated learning software "To save Taiwan" which attract the user to learn the microbiology 
knowledge and skills. This interactive software provides multimedia and random plots, which enable user to 
play the role in the story. It can also served as the tool to convey the right scientific attitude and social caring 
to learners. 




The evaluation of this study showed promising results. It is possible and valuable to adapt situated learning 
to other disciplines in higher education. A disciplinary can construct the learning process on a situated 



environment. By using the multimedia software, a learner can learn knowledge as well as the attitude in a 
near true story. He can then realize the meaning of the knowledge and identify himself with what he has 
learned and then applied to real environment. 
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The paper develops a topic of construction of the normative student model. The 
subject student model is a part of it representing a sum of demands to the 
curriculum of the subject, to students’ knowledge and skills, and a semantic 
model of the domain. The subject student models pick out the educational 
domains from all the multitude of the domains, so the subject model is a model 
of an educational domain. Examples in physics are given. 

Keywords: student modeling, domain modeling, knowledge, skills, semantics 



1 Introduction 

A fundamental concept of modem didactics and pedagogical psychology is the student (learner) model. It 
arose within computer technologies of education and was provoked by the necessity to formalize our 
representation about students. Of course such representations had been woiked out long before any 
appearance of computers, and definite formalization of them began together with didactics. But it is 
computer technologies that gave a new impulse to development of these representations, transformed them 
into an object of deep investigations, transferred to a qualitatively new level [8,9]. 

In the widest sense, the student model is our knowledge about a student. There are two sides here: (1) 
knowledge about how the student is, and (2) knowledge about how we want to see him/her. The first 
knowledge is determined by the way of analyses of student’ s behavior, and it is natural to call it a behavioral 
student model. It is changing together with the student’ s change therefore it is also called dynamic, or 
current, one. Mechanism of construction of this model is the cognitive diagnostics [9], 

Knowledge about how we want to see students, that is, demands to their final state is a normative student 
model. As a rule, this knowledge is various. It consists of demands to personal qualities of future specialists, 
their professional qualities and skills, their knowledge and skills in different subjects, characteristics of their 
physical and psychological state, and so on. The final aim of teaching is achievement of such a state when 
the behavioral student model concurs with the normative one. 



2 The subject student model 

A part of the normative student model determining domain knowledge is a subject student model [3]. In 
knowledge engineering, it is called expert knowledge, or domain model [5,6]. The subject student model 
picks out the educational domains from all the multitude of the domains, so the subject model is a model of 
an educational domain, or a model of a subject. Let us note that if the dynamic modeling is quite a 
developed branch of Artificial Intelligence, the domain modeling is developed to a lesser degree. It is clear, 
as specialists in Artificial Intelligence, as a rule, are not the ones in any other domain. 

Under knowledge they understand the main conformities to natural laws helping us to solve particular 
problems (production, scientific, economic, and others) [5]. Facts, concepts, algorithms, 
intercommunications, rules, strategies of making decisions, and so on make up knowledge. The pithy sense 
of the concept «knowledge» is that knowledge reflects our imagination about domains and expresses a 
system of concepts, as well as relationships and dependencies between them. 

According to the classification, there is a division of domain knowledge into declarative and procedural ones. 
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The first is statements about properties of the subjects of a domain and relationships between them. The 
declarative knowledge is often called a factual one, and this reflects its essence very well. The procedural 
knowledge describes the order and character of the transformation of the domain objects. Its another 
appellation is rules. In our opinion, it is not quite right, as the declarative knowledge, giving relationships 
between the objects, is also rules. Thus the procedural knowledge is not simply rules but rules of 
transformation. 

The final aim of instruction is formation of way of acting. The way of acting is realized via skills in the 
practical activity [7]. The mechanism of this is operation with knowledge (both the declarative and 
procedural) being displayed in the behavior of a person. Therefore, in a wide sense, skills are attributed to 
knowledge, namely behavioral one [9]. The procedural knowledge is realized in skills. And sometimes, 
skills are called the procedural knowledge but, as we could see, the term “procedural knowledge” has been 
already occupied. Definition “operation knowledge” reflects the essence of the things clearly and in the most 
unambiguous manner. Thus, the subject student model has to contain skills that are to be formed in the 
process of instruction. Let us call a list of such skills the operational subject student model. 

The declarative component of the domain knowledge makes up a semantic part of it, namely the semantic 
student model. 

One of the distinctive properties of knowledge is that it has a certain structure. It is very important, 
especially for the instructional material, to define its structure. It is well known that to master a portion of 
the instructional knowledge is to determine its place in the structure of the instructional material. Therefore, 
one of the problems while constructing the subject student model must be determination of the subject 
knowledge structure. Studying the structure of the instructional material is a theme of an independent 
important and deep investigation. The subject student model must give more or less extended ideas about 
what the subject knowledge is. Such ideas are an essential part of any curriculum. A usual way here is a 
thematic approach when themes are enumerated. Let us call a list of themes liable to studying the thematic 
subject student model. 

In teaching, it is very important methodologically to determine which role either knowledge plays and which 
functions it carries. In other words, it is necessary to fulfil a functional structuring of the instructional 
knowledge. It can be done with the help of a list of functional rubrics. The functional knowledge will be 
determined in such a way. Within it, there is knowledge performing both nontransforming functions (for 
example, facts, conclusions) and transforming ones (algorithms, methods, instructions). The functional 
knowledge makes up the functional subject student model. 

In such a way, we suggest a four component subject student model consisting of thematic, functional, 
operational, and semantic parts. Such a subject student model in physics is carried out at the physics and 
didactics of physics department of the Donetsk State University [1-3]. 



3 The thematic subject student model 

The thematic subject student model has been well known for a long time. In essence, it is a usual curriculum 
of the course, its program. It is built just according to the thematic principle, sections and themes liable to 
studying are enumerated in it. The model reflects the structure of the course. The program can be worked out 
in detail to different degree but it is always neither knowledge itself, nor its content but its names. In fact, 
this is a define characteristic of the subject knowledge, some knowledge about the subject knowledge. 
Knowledge about knowledge is called metaknowledge. Thus, the thematic subject student model is a 
metaknowledge. 

It is a natural and convenient model for planning and organizing the instructional process. The more, it is an 
obligatory normative document. Preparation of any course begins with its creation (that is, creation of the 
course curriculum). Nevertheless, it is excessively general to use it for diagnostics. 

As a rule, knowledge in many computer tutoring systems is structured according to the thematic student 
model. 



21 



4 The functional subject student model 



The functional subject student model shows which role either knowledge plays; and it is also 
metaknowledge. It has a define structure in the horizontal direction, which may be given with the help of 
some rubrics. The role of knowledge and its functions depend on a particular subject. For example, we 
picked out the following rubrics for physics courses ; concepts, wordings, laws, properties, consequences, 
conclusions, reasons, formulas, equations, models, methods, and algorithms [3], The rubrics have a filling 
that, nevertheless, also does not reflect semantic of the subject and is metaknowledge. 

It is the subject student model that allows working out in details what students must know. Let us give an 
example from the molecular physics. Students have to know: definitions of the concepts: mole, 
thermodynamic system, pressure, temperature, density, concentration, ideal gas, equation of state, and so on; 
wording and consequences of: Pascal’ s law. Maxwell’ s and Bolzmann s distributions, Kirchhoff s law, and so 
on; deductions of: the mine equation of kinetic theory, equation of the adiabatic process, law of atmospheres, 
and so on. 



5 The operation subject student model 

As it was noted, the operation subject student model is a list of skills liable to mastering by students. Let us 
note that skills in education make up a hierarchical system [2], It consists of five groups of skills: 
fundamental, methodological, general, inter-subject, subject. Subject skills take the highest position in the 
hierarchy of skills. 

We pick out three classes of the subject physical skills: general, particular, and experimental. The general 
skills are, on the first hand, methodological ones. Spectrum of the particular skills is far wider, for example, 
there are more than 200 them in the list in physics. According to the contents of the instructional material, 
the following skills are picked out: to find, to determine, to fix, to build, to obtain, to calculate, to compute, 
to estimate, to distinguish, to pick out, to sort, to take into account, to represent, to traverse, to decompose, 
to compose, to generalize, to put in practice, to use, to formulate. 

There is a fragment of the list of the skills below: 

3.1. General skills 

To analyze physical processes and phenomena, to estimate orders of physics magnitudes and determine 
essential factors, to build physical models, to build mathematical models of particular physical processes 
and phenomena, to determine boundaries of applicability of the models, and so on. 

3.2. Particular skills 

3.2.2. Molecular Physics and Thermodynamics 

To estimate quantity of particles and their mass in particular conditions, to determine parameters of state of 
gas, to determine number of degrees of freedom and molecular mass of a gas and mixture of gases, to 
determine possibility of the use of the model of an ideal gas, to make use functions of distribution to find 
average values of physics magnitudes, and so on. 

Experimental skills are divided into three groups: to measure physical magnitudes ; to reproduce 
independently physical phenomena and processes; experimental particular skills. 

There is a hierarchical structure of the subject skills corresponding to the development of the subject in 
instruction. Besides that all of them also have a definite structure in the horizontal dimension because they 
are complicated, or composed, skills. In order to master them, a wide spectrum of skills both of the lower 
levels and subject is necessary. For example, skill to solve physical problems is composed of ten simpler 
skills : to pick out the necessary information from the condition of a problem to solve it, to code the condition 
of the problem in a word form, to draw a picture to the problem, to choice a rational method of solving , and 
so on. 



6 The semantic subject student model 

Semantic knowledge in different subjects is contained in textbooks, other training literature. There are two 
parts in the content of any textbook: CON-1 and CON -2 [7]. CON-1 is knowledge making up the content of 
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a domain directly, CON-2 is knowledge attending the CON-1 (for example, knowledge from other subjects, 
interpretations, explanations, examples from life). In fact, it is the CON-1 that is the semantic knowledge of 
a domain. Nevertheless, this knowledge is not picked out especially, it is distributed all around the textbook, 
interacts with another knowledge, and is not formalized. 

Semantic knowledge represents the declarative component of the subject knowledge as the procedural 
knowledge is realized in skills (operational knowledge). Thus to construct a semantic student model on the 
basis of a textbook, it is necessary to pick out domain facts from it and group them in a definite order. 
According to their structure, facts may be of a great variety. As a rule, they are compound ones. Nevertheless, 
elementary facts may be picked out that, appearing in different relationships, form the compound facts. 
General questions of representation of facts in instruction are considered in works [4]. For example, 
expression “ Translational motion is the motion that all the point of a solid body have identical trajectory * is 
a compound fact as it can be represented as a set of the following elementary facts: (1 ) a solid body moves ; 
(2) all the point of the body have identical trajectory; some motion is called the translational one. 

One can easily see that the elementary facts do not carry any semantic loading of the domain although they 
contain domain terms. Only on gathering together in a compound fact they acquire some doma in sense. Such 
compound facts are finished thoughts and they are represented by finished sentences, or expressions. Let us 
call them the semantic facts. As a matter of fact, the semantic facts are a unit of the domain knowledge, as 
smaller portions of it have no domain sense. The objects of the expression are concepts, phenomena, 
processes, laws, principles, theorems, conclusions, consequences, reasons, properties, rules, and so on. 

It is the full set of the semantic facts that is the semantic subject student model. The order of their disposition 
is subordinated to the logic of the development of the course. 

Such a semantic subject student model was firstly constructed in Gas Dynamics and than in Physics [1]. 
Those were very small brochures because there were no calculations, proofs, and explanation in them. 
Nevertheless, they contained all the statements of the courses. These brochures received the title semantic 
synopsis. As an example, there is a fragment from a physics semantic synopsis below: 

3.1. The elementary work of a force is defined as the scalar product of the elementary displacement 
of the point of the force application. 

3.2. The work of a force is defined as a line integral from the elementary work along the trajectory of 
the point of the force application. 

3.3. The unit of the work is one joule that is equal to a work done by a force of one newton on a 
displacement of one meter. 

In the opinion of instructors and students, the synopsis turned out an effective means while consolidating the 
instructional material, preparing to seminars. It helps to size up the structure of the instructional material, 
pick out and easily memorize the most essential its moments. It is very important that student remember 
them for a longer time. 

The synopsis allows carrying out fast and regular control students’ knowledge during a lecture. In this case, 
the expressions serve as a base for the open type test tasks being created by missing some keywords in the 
expressions. Students note a great value of the synopsis while preparing to the examinations when there is a 
danger do not pick out and master the main statements of the course. 

Let us note that the semantic facts are distinctive rules as they define character of relationships between the 
elementary facts. In other words, they are rules according to which the elementary facts are connected 
between themselves. This circumstance stipulates possibility to represent the semantic knowledge by means 
of the production method. It is done with the help of rules of a kind “if A than B” where A and B are some 
facts. An example of such a representation of the above mentioned definition of the translation motion is 
given below: 

If < a solid body moves> and 

< all the point of the body have identical trajectory> 
than <such a motion is called the translational one>. 

Each of the expressions may be represented in such a way. Thus the production knowledge base of the 
subject will be constructed. Its constructing is considered in details in work [4]. As our practice shows, 
constructing production knowledge bases by students while learning is an effective kind of learning activity. 
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7 Conclusion 



An approach to construction of the subject student model as a part of the normative one is described. The 
model consists of four components: thematic, functional, operation, and semantic. The thematic model gives 
ideas about the structure of the subject, the semantic one reflects its content, functional one determine what 
students have to know, and operation one does what students have to be able. The approach allows 
constructing more detailed current student models and reaching the main aim of teaching, namely forming 
the way of acting, more successfully. 
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Information technology integrated instruction is the education tendency in the 
future, and it is also an important issue in the development of education in 
Taiwan. An assessment framework is needed during experimenting with 
integrated instruction and when integrated instruction is officially implemented. 

The framework can help us understand the implementation and provide 
information for future reference. This article proposes an framework for 
assessing information technology integrated instruction. The framework 
includes kernel and periphery parts. Kernel part refers to the whole teaching 
process, including information technology, curricula, learning materials, 
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to the surroundings situation, including teachers, students, information 
specialists, administrators, classroom settings, computer laboratories, campus 
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1 Introduction 

The rapid development of information technology (IT) has not only brought about major effect on economy 
and industry but also made a great impact on society and education. In particular, the prevalent use of 
computers and the rapid development of the Internet have gradually changed our life style and pattern, with 
their impact on education being unprecedented. Many educators and policy makers believe that technology 
can be a catalyst for educational reform [3,4,10]. They suggest that the use of technology in classrooms will 
shift the roles of teachers and students. Teachers will act more like facilitators by helping students access 
information, process it, and communicate their understanding [4]. 

Beginning the 2001 academic year, Taiwan will implement phase-by-phase the nine-year integrated 
curriculum for its elementary and junior high schools [11]. To cultivate students* basic ability to “apply 
technology and information”, the new curriculum will have to emphasize integrating IT into the teaching of 
various courses. Amid this major reform of curriculum, the Computer Center of the Ministry of Education 
has planned for the integration of information curriculum with other areas of learning [7]. At the same time, 
it has selected 18 elementary and junior high schools in which teaching experimentations will be carried out 
[1]. Therefore, an assessment framework is needed during experimenting with integrated instruction and 
when integrated instruction is officially implemented. The framework can help us understand the 
implementation and provide information for future reference. 



2 The essence of information technology integrated instruction 

The United States has implemented IT integrated instruction for years. Many educators are now actively 
using technology along with effective teaching strategies to integrate technology into their curriculum [9]. In 
contrast, IT integrated instruction is still a newly heard noun in Taiwan. Many teachers are unfamiliar with it, 
and some think of it as another name for computer-assisted instruction (CAI). Information technology has 



developed rapidly, and the role of IT in education has changed over these years, from being an auxiliary to 
teaching to being an indispensable tool of education. Therefore, IT integrated instruction is distinguished 
from CAI. 

In IT integrated instruction, information technology is an indispensable tool in the teaching environment 
because it is integrated into the curriculum, learning materials, teaching and learning [2], Moreover, the 
traditional curriculum, materials, and teaching are transformed through the characteristics of information 
technology: the subject-based curriculum and materials become student -based; the teacher-driven teaching 
activities become student -centered. Information technology is integrated when it is used in a seamless 
manner to support and extend curriculum objectives and to engage students in meaningful learning. It is not 
something one does separately; it is part of the daily activities taking place in the classroom [3]. 




F 

igure 1. The assessment framework for information technology integrated instruction 

Figure 1 depicts the assessment framework of IT integrated instruction. The assessment framework consists 
of two major parts: Kernel Part and Periphery Part. The kernel part primarily assesses the whole teaching 
process. Because the implementation of IT integrated instruction will bring about changes to teaching, the 
aspects to be assessed in this part should include not only the use of IT in teaching but also other 
perspectives of teaching: curricula, learning materials, instruction strategies, learning activities, and 
evaluation. The periphery part primarily assesses the teaching environment, learning resources, information 
equipment, personnel qualities, and administrative as well as professional support. All these factors will 
influence the outcome of teaching. In particular, IT integrated instruction is in need of supportive and 
coordinated environmental conditions. There are many perspectives of the periphery part that are related 
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with IT integrated instruction, and ten of them are carefully identified and included for assessment: teachers, 
students, information specialists, administrators, classroom settings, computer laboratories, campus 
instruction network, Internet, digital materials, and instruction/leaming software. 



3 Assessing the kernel part 

The kernel part refers to the whole teaching process, and Table 1 shows the perspectives and emphases to be 
assessed. The aspects of the kernel part are illustrated in the following paragraph. 



Perspective 

Information technology 
Curricula 

Learning materials 
Instructional strategies 
Learning activities 
Evaluation 



Table 1 . Emphases of the kernel part to be assessed 

Emphasis 

The use and role in instruction 

Subject-based separate curricula or Student-centered integrated curricula 
Sequential or problem-based 

Traditional expository approach or constructivist approach 
Teacher-driven or student-centered 

Traditional paper-and-pencil testing or multiple assessment 



3.1 Information Technology 



Information technology may refer to equipment or products, such as computers, network, peripherals, etc. It 
may also refer to the methods or processes in which the equipment of IT is used to help with the solution of 
problems. It is the purpose of implementing IT integrated instruction not only to enable students to use the 
equipment or products of IT but also to use the IT equipment to solve practical problems in learning and life. 



In this perspective, we care about how IT is used in teaching and what role IT plays in teaching. The level of 
the use and role in instruction is developed: 

• Nil (level 0)* IT is not used and plays no part in teaching. 

• Isolation (level 1): IT is used to teach students how to use IT (e.g. keyboarding, drill-and-practice, 
word -processing activities). There is no or little connection between IT and instruction content. 

• Supplement (level 2): Teachers use IT to assist instruction and students use IT to aid learning 
occasionally. IT is viewed as a supplement to existing instructional program. 

• Support (level 3): IT is needed to complete most learning activities. IT serves as a support to 
instruction. 

• Integration (level 4): Students and teachers can use IT in every-day leaming/teaching naturally, 
confidently, and actively. IT is expansively viewed as tool, process, method to find solutions to 
authentic problems in any time anywhere. 



3.2 Curricula 



For elementary and junior high schools, the curricular idea should be life -centered and be in compatible with 
the development process of students’ physical and mental abilities; respect character development, inspire 
individual potential; cultivate civic qualities, respect the value diversified culture system; enhance science 
knowledge and skill, meet the requirements of modem life. The design of curriculum should be based on 
students, on practical experience, and devoted to cultivating the basic abilities required of modem citizens 
[11]. Therefore, the curriculum should be designed as student-centered integrated or interdisciplinary 
curriculum, not subject-centered separate curriculum. 

IT is used as a tool to help students solve the problem. IT literacy should not be taught as an isolated subject, 
nor should activities with IT be isolated from other activities in the classroom [12]. Therefore, Taking the 
students to the computer lab once a week for 40 ninutes is not necessarily integration [3]. The teachers 
should commit to designing student-centered integrated curriculum and integrate IT into the curriculum. 

33 Learning Materials 

Textbooks are the main materials for elementary and junior high schools and the primary learning materials 
of students. In traditional education, textbooks were unified, having only one version. They were based on 
subject systems and separate from students’ living experience. Besides, it was difficult to innovate them, 
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they could not meet society’ s requirements for rapid transformation. In 1996 Taiwan implemented a policy 
which would partially allow publishers to edit and provide textbooks for elementary and junior high school 
so long as they are approved by the Ministry of Education. On February 3, 1999, VIII (2) of National 
Education Act was empowered, which unequivocally directs the full use of ministry -approved textbooks for 
elementary and junior high schools. A new epoch for textbooks was thus heralded in. Teachers should be 
able to exert their professional autonomy, and students should be allowed a flexible, autonomous learning 
leeway. 

The presentation of learning materials should not be limited to static traditional books, but the characteristics 
of computer multi-media should be used to present these materials. Static words and pictures, animated 
pictures and films, voice, acoustic effect and music in combination would make teaching materials lively 
and motivate students to learn. Besides, they can help students to understand abstract concepts or knowledge 
and enhance learning effectiveness. If hyperlink technology is used, nonlinear learning materials can be 
designed so that what students see can be highly individual and not the same. As such, the content of 
learning materials is flexible, adaptive to individual difference, and compatible with the spirit of 
individualized learning. 

In addition to textbooks, there are many resources on the Internet that can be used as learning materials. 
These resources can provide “instant”, “living” information [8]. Teachers not only can search for 
information to be included in teaching materials but also can use the real-time information on the Internet to 
conduct teaching. Students not only can search for information on the Internet but also can conduct 
independent learning any time, any place by using the learning materials on the Internet. 

The use of information technology can make learning materials diversified and lively, make their content 
flexible and integrated with life. Not only can teachers easily motivate students to learn, but students also 
can learn happily in a rich teaching situation. 

3.4 Instruction Strategies 

The teaching strategy of the traditional expository approach is teacher-centered. Students learn what is 
taught by the teacher, but are given a limited room for thinking, discussion, presentation and exploration. 
The teaching effectiveness is ostensibly good, and students’ performance on examination seems impressive. 
Yet this approach contradicts the essence of education. In a series of meaningless learning process, what 
students learn is segmented memory that is extraneous to their experience and cannot be applied in practical 
situations of their daily life. Nor can they enjoy learning. 

Constructive teaching strategy is student-centered. The teacher would first arrange a teaching situation to 
arouse students’ motivation for learning and then would conduce students to explore and think. Through the 
interaction with peers, the students can gradually integrate the new knowledge into their own system of 
knowledge and make it an essential part of this system. By this constructive teaching, students must actively 
learn, while the teacher can only play the role of facilitator, auxiliary, and consultant. 

When students use teaching software and browse for Internet resources, they must explore and think actively 
and construct their own knowledge through the interaction between machine and person and through social 
interaction. Therefore, the teacher is a “coach” for the student rather than a provider of knowledge. Self- 
directed learning is an attainable goal for both the student and teacher when IT is integrated in the various 
content areas [6]. That is, IT integration is most likely to occur in learner-centered classrooms in which the 
teacher acts as a facilitator [3]. 

3.5 Learning Activities 

Traditional lecture-based and teacher-driven activities can no longer satisfy the needs of modem education. 
It is not only monotone, also lacks interaction between peers. Learning activities should be student-centered 
so that the learner can actively work to explore knowledge, clarify concepts, and gradually construct his/her 
system of knowledge. In addition, project-based and cooperative learning activities should be adopted to 
allow the learner the opportunity to produce high-level interaction with his/her peers. These activities not 
only can cultivate a respectful, responsible, and confident attitude and the abilities to express, communicate, 
coordinate, think, and create but also can increase learning effectiveness. 

In cooperative learning activities, students can use computer to communicate and discuss, or use a certain 
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support cooperative work software to facilitate collaboration. Finally, multi-media would be used to present 
the learning effectiveness of students. Cooperative learning is not limited in local class. It can also be 
applied across schools, countries, and culture. Therefore, IT enriches the learning activity. 

3.6 Evaluation 

The traditional evaluation approach primarily depends on paper examinations and determines learning 
outcome by the scores on the test sheets. This type of evaluation measures only a dimension of knowledge, 
unable to reflect the wide spectrum of learning process. Future evaluation will become diversified; 
performance evaluation may be conducted along with paper evaluation; students’ self-evaluation, peer 
evaluation and juried evaluation may be conducted along with teacher’ s evaluation; in addition to evaluating 
learning outcome, the learning process should be evaluated; in addition to quantitative evaluation, qualitative 
evaluation should be adopted; in addition to evaluating cognitive domain, the evaluation of affective and 
skill areas should be included. Only such a comprehensive evaluation can reflect the learning process, not 
only be able to understand what the student has learned but also be able to understand how the learning has 
occurred. 

IT integrated instruction is helpful to the implementation of diversified evaluation. For example, the 
electronic portfolio is an ideal means of integrating IT into the instruction. It gives the student and teacher an 
alternative form of assessment. Furthermore, electronic portfolios motivate students to produce quality work, 
and they also increase students’ self-esteem by showcasing their best work [6]. 



4 Assessing the periphery part 

The periphery part primarily refers to the surrounding situations. Table 2 shows the perspectives and 
emphases to be assessed. The following illustration is based on perspectives. 

4.1 Teachers 

The teacher is vital in leading teaching activities. Without sufficient information literacy and professional 
ability, he or she cannot apply information technology on teaching, let alone implement IT integrated 
instruction. Regarding professional ability, the teacher should be able to integrate IT, in addition to assessing 
software and digital materials. The attitude is another emphasis of assessment. If the teacher has a positive 
attitude toward computer, he/she can readily introduce and apply computer on teaching; if the teacher can 
accept the change in teaching status and role, the implementation of IT integrated instruction would not 
cause a great impact. 

4.2 Students 

Students are the chief character in education. In teaching, students should take the initiative to construct their 
own knowledge. In implementing IT integrated instruction, students can obtain from the process related 
knowledge and skill and steadily strengthen their information disposition. Gradually students should be able 
to use, naturally and confidently, computer equipment in active learning and to construct their system of 
knowledge. 

4.3 Information specialists 

Teachers are not information specialists. In extensive application of IT to teaching, they will definitely 
encounter many technical problems that can not be solved by them. In this case, information specialists can 
support teachers in solving such problems. It is much easier for information specialists with education 
background to integrate IT with education and guide classroom teachers to implement IT integrated 
instruction. 

4.4 Administrators 

Whether administrators feel important about IT integrated instruction is intimately related with the 
implementation of IT integrated instruction. In addition, if he classroom teacher can gain sufficient 
administrative support, he or she will be more willing to implement IT integrated instruction. 



29 



4.5 Classroom Settings 



Generally speaking, teaching activities are conducted indoors. Therefore, the IT equipment in classroom is 
indispensable to the integration of IT into teaching [14]. The computer and peripherals should not be 
outdated. The operation system and application software installed in the computer should be appropriate for 
the use by students and suit the needs of teaching. Moreover, for a class of more than 10 students, a large 
display device or broadcasting teaching equipment is needed. Finally, it matters whether they are managed 
properly or whether the fair use by students is ensured. 



Table 2. Emphases of the periphery part to be assessed 



Perspective 


Emphasis 


Teachers 


Information literacy and professional competency 

Attitude toward information technology and instructional change 


Students 


Information literacy 

Attitude toward information technology 


Information Specialists 


Professional competency 
Support for teacher 


Administrators 


Attitude toward information technology integrated instruction 
Support for teacher 


Classroom Settings 


Number of computers and person-machine ratio 
Grades and fixtures of computer 
Operating system and application software. 
Peripherals (e.g. printer, scanner, digital camera) 
Broadcasting teaching facilities 
Management 


Computer Laboratories 


Number of computer labs, number of computers and person-machine 



ratio 

Grades and fixtures of computer 
Operating system and application software. 
Peripheral (e.g. printer, scanner, digital camera) 
Broadcasting teaching system 
Management 



Campus Instruction Network 


Structure of campus network and network type 

Domain account 

File server and database server 

CD cabinet (perhaps made possible through software simulation) 


Internet 


Method and speed of Internet connection 
Actual connection speed 

Internet server (e.g. web server, proxy server, DNS server, mail server) 
The mechanism to filter out inappropriate information. 


Digital Materials 


Digital materials that can be used on the Internet 
Digital materials created by the teacher 


Instruction/Leaming Software 


Quantity 

Adaptation 



4.6 Computer Laboratories 

In a situation in which IT is integrated into teaching, sometimes it is required that one person have one 
machine. Computer laboratories can justly meet this requirement. Therefore, the management of computer 
laboratories is an important assessment item and can decide whether the computer equipment can 
sufficiently support classroom teachers [14]. Moreover, computer laboratories can also provide the most 
appropriate places for teachers’ advancement and students’ training of information skill. The equipment in 
the computer laboratories should not be outdated. Furthermore, there must be a broadcasting system, 
enabling students to know the whole content of teacher’ s lecture in a ready manner. 

4.7 Campus Instruction Network 

The planning and erection of campus instruction network aims not only to construct an instruction network 
on campus but also to enable every classroom on campus to connect to the Internet through the campus 
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network. After the campus network is erected, File Server and CD cabinet (perhaps made possible through 
software simulation) should be erected, in which the teaching software owned by the school is stored so that 
all the teachers of the school can access to it readily and can apply it to teaching. In addition, the 
establishment and management of network account is equally important, ensuring the safety of information 
[14]. 

4.8 Internet 

There are unlimited, un-exhaustive teaching resources on the Internet. If computers can not be connected to 
the Internet, the application of IT to teaching is compromised. Therefore, it is very important to provide 
information settings of the Internet. In addition, it is needed to erect Internet-related Server, in particular, 
Web Server must be erected so that teachers’ teaching information and the learning outcome of students can 
be stored on it to facilitate examination and simulation by teachers and students. Besides, Internet is full of 
erotic and violent information which is unsuitable to students. It is extremely important to build a 
mechanism to prevent students from seeing those inappropriate content [14]. 

4.9 Digital Materials 

Digital materials may be presented through information equipment and be used in teaching activities. 
Therefore, plentiful digital materials should be able to help integrate IT into teaching of various subjects. 
Therefore many on-line materials on the Internet can be used for teaching purpose. To decrease the amount 
of time required for browsing and facilitate the use of the materials by teachers and students, on-line index 
and search systems are also required. In addition, if on-line materials are not appropriate for teaching needs, 
classroom teachers may create their own materials to be presented on information equipment or use 
materials created by colleague teachers with the approval from the original designer [14]. 

4.10 Instruction/Leaming Software 

Computer Assisted Instruction (CAI) and Computer Assisted Learning (CAL) software is a help to teaching 
and learning. With more software, teachers are better equipped to apply IT to teaching. This software should 
be stored on the CD cabinet or File server on the campus network so that teachers can readily use it 
whenever needed. In addition, if existing teaching software available on campus is evaluated, further 
information can be provided to teachers [14]. 



5 Conclusions 

That teachers and students can extensively use computers for teaching or learning purpose to heighten 
teaching qualities and learning effectiveness is the ultimate goal of the infrastructure construction of 
information education [5]. In other words, integrating computer into teaching of various subjects is the 
ultimate goal of the Ministry of Education in promoting information teaching [13]. What IT integrated 
instruction means is not merely to assist teaching by computer but work to integrate IT into curriculum, 
learning material and learning activities. At this point, the role of teachers begins to transform, from that of a 
main character to that of a support character. Therefore, the implementation of IT integrated instruction not 
only harmonizes with the u ltimate goal of information education but also prompt the reform of education so 
that learning becomes more effective, efficient, and meaningful. 

IT cannot be successfully integrated overnight. It needs to take years to complete the process. The process 
should be carried out in order, stage by stage. Taiwan’ s IT integrated instruction is germinating. The 
assessment framework set forth in this article can be used not only to carry out practical evaluation but also 
serve as reference for development. Teachers’ in-service education, pre-service training, administrative 
support, enriching IT equipment, developing appropriate digital materials and teaching software should be 
taken to strengthen the perspectives of the periphery part and to diversify the surroundings so that teachers 
can realize the benefits brought about by IT on education, Accordingly, teachers can apply IT to teaching, 
gradually infuse IT into learning activities, curricula, learning materials, and adopt student-centered teaching 
strategies and multi-facet evaluation. All this can lead to the fulfillment of the meanings of IT integrated 
instruction. 
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This paper reports an effectiveness research of e-mail discussion review support 
system with summary extraction method. The support system we have 
developed can automatically extract summary sentences from the normal 
conversational style language in e-mail messages using reference relationship of 
e-mails that participants have discussed. One could use the summary sentences 
for looking back on discussion, and use them as an idea database at a glance. 

Japanese natural language processing technology has been applied in the 
proposed method. In order to evaluate the effectiveness of the system, we 
conducted experiments using a questionnaire and protocol analysis. We 
compared the two system; the system with and without summary sentences in 
the table of e-mail content. As a result, following fact-findings were obtained. 

The system with summary sentences could promote reading strategy such as 
utilization of table of contents and comprehension of e-mail message structures. 

On the other hand, the system without summary sentence makes the reader pay 
attention to the detail information such as name of discussing member. 

Finally, we concluded that the system with summary sentence is effective for 
understanding of relationship among various e-mail messages. 

Keywords: Mailing lists, Natural language processing, Distance learning, 

Learning environment, Summary sentence extraction, Collaborative 
learning, Factor analysis, Reading strategies 

1 Introduction 

Collaborative learning support environments for network-based discussion appear to be investigated quite 
often [ 1 ] [2]. For instance, e-mail is extensively used in the classes for learners’ communication. 

The research topic we reported here is collaboration support tools that intended for e-mail discussion. For 
the purpose of sharing of participants’ activities on computer networks, we have proposed a summary 
extraction method along the development of mailing list discussions and an outline presentation tool for 
mailing list [3][4][7][9]. Japanese morphemes analysis system [8] is applied in our researches. This web- 
based tool supports reviewing the past discussion on the mailing list. As for results of the summary 
extraction method, we conducted comparative evaluation between the result of human summarization and of 
the method. The result suggests that the proposed method can detect major sentences in e-mail articles 
properly [4]. 

There is a number of preceding researches on the keyword and summary sentences extraction methods of 
documents [5][6][15]. But the most of extraction methods in preceding researches applied to well- 
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documented text, like the newspaper manuscript or research paper. On the other hand, this research targets 
on the conversational style language in text form. For identifying the outline of e-mail discussion, there are 
many difficult problems in e-mail messages. These are: 

• E-mail messages are conversational style language and many summary extraction methods using 
syntactic information could be not applied. 

• The title of e-mail might not be changed as the discussion continues, if so, the title is not meaningful as 
the summary of documents. 

• The method should identify the flow of discussion corresponding with e-mails in order to grasp the 
topic. 

Besides, most of evaluation experiments in summary extraction method with natural language processing 
technology focus on the validity of algorithm, like adaptability or reproducibility. About analysis of reading 
comprehension when additional information, e.g. summary, is given, we could refer Ausubel’s research on 
the advanced organizer model [10], The paucity of reports on sentence comprehension process encouraged 
us to investigate it. 

The purpose of the present paper is to analyze how the summary sentences accomplishes to an actual 
comprehension process. In this paper we describe an experimental study of e-mail message reading process 
with or without the extracted summary sentences. 

In the first experiment, we investigated e-mail message reading strategies using responses of questionnaire. 
We conducted comprehension test and reading process analysis. In the reading process analysis, the result 
was divided into seven factors using factor analysis. The system with summary sentences could promote 
reading strategy such as utilization of table of contents and comprehension of e-mail message structures. On 
the other hand, the system without summary sentence makes the reader give attention to the detail 
information such as names of participating members. 

In the second experiment, we analyzed peer discussion processes while reading e-mails on the World Wide 
Web (WWW) interface. We conducted the comprehension test of the e-mail messages. We also conducted 
protocol analysis of e-mail reading comprehension. Also hereupon, we compared the results with two 
conditions, one is a group to which the summary sentence of the e-mail messages was given, and the other is 
a group without summary sentence of e-mail messages. The results of protocol analysis show some 
difference in the number of utterance collected during the experiment. 

2 Summary extraction method along development of discussion 

The summary extraction method was 
discussed in our preceding research 
[3] [4]. In this paper, we de-scribe the 
outline of the extraction method for 
better understanding by the readers. 

2.1 Idea of the extraction 
method 

We tried the extraction of keywords 
and summary sentences of the 
discussion from the document in the 
mailing list based on the preceding 
research [11] intended for the 
discussion such as Netnews. This key- 
word extraction method can be used in 
the discussion environment with the 
following features; (1) The change in 
the topic does not take place easily in a 
row. (2) There is a habitual practice 
that the participants do repeated 
revisions during the discussion, and 



Flow of discussion 




Phase 2: Select keywords for summary (S») by set operation 



c. _ c. r /e. i c.i 




Set St means the newly appeared 
nouns in the target message, and 
repeatedly appeared in the 
response message. 



Phase 3: Select sentence(s) include elem ent(s) of S,. as “summary". 



Figure 1: Flow of Summary Extraction Procedure 
from the target message. 
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uses the quotation appropriately. But although it is limited in our case, e-mail discussion might develop in 
many ways, and the topic is changeable. The relationship of e-mail message for the keyword extraction 
between the target message and the past messages is little in e-mail discussion. 

Then, in this paper, we set up a hypothesis: Although there was a dependency on the topic, e-mail messages 
with new information are tempted to encourage responses later. That is, we can treat them as topic making 
messages in the mailing list. We proposed a summary extraction method that enables pick up those new 
information as keywords and summary sentences in the messages [3] [4]. Figure 1 shows flow of keyword 
and summary extraction by this method from the content of the message of the mailing list. 

However, this summary extraction method supposes both preceding and response messages must be 
consecutive in the thread. Therefore, we set some assumptions for these exceptions. When the target mes- 
sage is the beginning message in the thread, the title of the message is also used and extracts common nouns 
among the title of the target message and the body of related messages. On the other hand, when the target 
message is the last message in the thread, we choose keywords only from the preceding and the target 
message, and common nouns in both messages is treated as keywords for the target message. Moreover, 
summary sentences are regenerated when there is a new message in the mailing list. 

2.2 Summary generation and WWW display tool 

We implemented summary generation and display tool using the proposed summary extraction method. This 
can be operated on the World Wide Web (WWW) to refer to past messages of mailing list [7]. Fig-ure 2 
shows, the display of Web page with and without summary sentences. These Web pages fulfill the role of 
table of contents (TOC) of mailing list. Readers look for contents from the list view with tree structure along 
continuity of e-mails. They can trace the body of each message from Web link. TOC shows serial number, 
writer, date of issue, and the title of the e-mail. In Figure 2(b), under the link to the body, summary sentence 
obtained by the noun set is displayed. When more than one sentence is extracted by the method, it becomes 
so complicated that the implication of TOC is diminished. So we referred to the procedure widely used in 
full-text search system [6], the number of displayed sentence is trimmed off to only one sentence that 
include maximum different number of chosen keywords. We treat that sentence as important sentence for 
TOC. 



3 Evaluation experiment in the e-mail message comprehension 

In this research, we carried out the evaluation experiment on effects of summary presentation while reading 
past e-mails on the mailing list. We conducted reading comprehension test and factor analysis of reading 
strategies. 

3.1 Methods 



3.1.1 Subsubsections 



In the experiment, we made the settings resembling the actual Web-based environment of the mailing list. 
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B6'ST COPY AVAILABLE 



We printed out the several e-mail messages in a row, referred to as “thread”, and the table of contents (TOC) 
for the e-mail messages in addition. Figure 3 shows the part of the experimental materials. To the semblance 



[NGP:01768] 

From: satotaka@*******“““*. ne.jp at 99/06/10 20:51: 
? ? 

? 

? 

[NGP:01770] Re: 

From: tetsuya@*******“Md at 99/06/10 21:22:02 

? 

[NGP:01773] Re: 

From: syun@*************.jp at 99/06/11 08:17:56 

? ' ? 



[NGP:01 768J 

From: satotaka@***********“*. ne.jp at 99/06/10 20:51: 



[NGP:01770J Re: 

From: tetsuya@***““***.id at 99/06/10 21:22:02 



[NGP:01773J Re: 

From: syun@*************.jp at 99/06/11 08:17:56 



(a) Printed TOC with summary sentences. (b) Printed TOC without summary sentences. 

Figure 3: The part of the printed experimental materials 



of Figure 2, the summary is generated from the proposed summary extraction method. It appeared in parallel 
beneath each entry in the TOC, or not appeared. E-mail messages for the summary extraction method consist 
of nine messages of mailing lists. The topic in the mailing list is the educational use of the Internet for 
foreign Japanese schools and domestic schools. 



3.1.2 Procedures 



Subjects of the experiment are 56 undergraduate students. None of the subjects know about the mailing list. 
The printed TOC as described above is affixed in front of the e-mail messages. The printed experimental 
materials were distributed to the subjects, and the researcher explained the experimental setting: “We are 
going to try to read past e-mails, and catch up with the exchange of the e-mail discussion.” In addition, the 
participants were asked to use TOC positively. 

The subjects read these documents for eight minutes. After the eight minutes, the researcher confirmed all 
the subjects had read the documents once. After that, the subjects were not allowed to read the docu-ments 
again, and they did the e-mail comprehension test. They had answered the following questions: 

1 . Write down the name of places which had appeared on the first e-mail as much as you remember. 

2. Write down the episode of the first e-mail as much as you remember. 

Later, they answered a questionnaire, which was consisting of 28 items with five-point rating scale and 
space for writing comments. The items were concerning the e-mail reading strategies. In order to make 
questionnaire, we referred the preceding research about sentence intelligibility [12] and our preceding 
researches. 



3.1.3 Experimental Design 

The factor of the experiment materials is presence of summary sentences in the TOC. We can divide the 
subjects into two levels. 56 subjects were randomly assigned to both two experimental settings of the 
materials, and were divided into the two groups of 28. Therefore, it is a between -subject experimental design 
with one factor. 



3.2 Results 



3.2.1 The comprehension test 

In the question 1: “the name of places 
which had appeared in first e-mail”, we 
compared the numbers of correct 
answers between two groups. We leave 
non-response persons off from the 
analysis. As a result of ANOVA, there 
was no significant difference in the 
number of correct answers 



Table 1 : Extracted factors and results of ANOVA 



Name 


Ratio 




1 . Read the content in detail and memorize 


21.5% 


**p<.01 


2. Use Table of Contents 


10.5% 


*p<.05 


3. Think about the development of discussion 


8.7% 




4. Combine their knowledge 


7.1% 




5. Think about the theme of discussion 


5.1% 




6. Read back and force 


4.3% 




7. Write down a memo 


3.9% 




Accumulated Explanation Ratio 


61.1% 
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(F(l ,48)= 1 .065, p>. 10). 

In the question 2: “the episode of the first e-mail”, 
we have chosen eight words from the message as 
answer words of the question beforehand. We 
compared the numbers of appeared answer words 
between two groups. We also leave non-response 
persons off from the analysis. As a result of 
ANOVA, there was no significant difference in the 
average number of the answer words (F(l,48)=.415, 

p>.10). 

3.2.2 E-mail reading strategies 



Table 2: ANOVA for Factor Scores 



(a) ANOVA for Factor Score of factor 1. 



Source 


: SS df MS F R 3 


Factor 

Error 


5.61 1 5.61 7.21 0.0098" 

38.11 49 0.78 


Total 


43.72 50 


(b) ANOVA for Factor Score of factor 2. 


Source 


SS df MS F R 2 


Factor 

Error 


4.62 1 4.62 5.99 0.0180* 

37.78 49 0.77 


Total 


42.39 50 



~p<0 



*p<.0 



The factor analysis with major factor method and varimax rotation method was applied to the 28 ques- 
tionnaire items concerning strategies of the comprehension for e-mail messages. 



As we shown in Table 1, we sequentially named the seven factors. We extracted these factors from the 
change in the eigenvalue. The accumulated factor explanation ratio was 61.1%. Next, factor score of seven 
factors was calculated par subjects. 

Table 2 shows results of ANOVA. As a result of ANOVA for seven factors, a significant difference was 
found in the first factor “Read the content in detail and memorize” (F(l,50) =7.212, p<.01) and the second 
factor “Use Table of Contents” (F(l,50)=5.988, p< 05). 



In addition, we compare the score for each item in two groups. 



As a result, the group with summary sentences could promote reading strategies such as “Usefully reading 
TOC help me to know the content of sentences” (t(51)=3.58, p<.01), and “Refer TOC to read the content in 
the messages” (t(52)=2.76, p<.01). Those who use summary sentences would have tendency that they try to 
know the relation between the content and the whole structure of the thread. 



On the other hand, the group without summary sentences would take reading strategies such as “Pay 
attention to the participant’s name or the name of places appeared on the e-mail while reading” 
(t(50)=2.34,p<.05), “Read the content carefully and memorize in detail” (t(51)=1.94, p<.10). Thus, they 
attempted to give attention to the detail information such as names of discussing members. 

3.3 Summary of the experiment 

In the experiment, there was a significant difference in the e-mail reading strategies while there was no 
significant difference in the recognition of e-mail contents. Our proposed method is a kind of new infor- 
mation presentation method for the support of e-mail reference. We might say our summary extraction 
method and display tool for mailing list could help readers to suppress consideration of detail information in 
the documents. On the other hand, these supports help to maintain the particular contents easier. 



4 Protocol analysis of e-mail reading process 

From the suggestion in the preceding section, adding summary sentences possibly provide a hint on the e- 
mail reading strategies. In this section, we examined changes of e-mail reading strategies when having the 
benefit of summary sentences using protocol analysis. To put it concretely, the subjects answer questions 
after reading the content of e-mail messages that is displayed on the WWW pages. We have observed the e- 
mail reading strategies while participants were reading e-mail messages. 

4.1. Methods 

4.1.1 Experimental materials 

We have used 43 e-mail messages of the mailing list for the summary extraction method. Educational use of 
the Internet in foreign Japanese schools and domestic schools was focused in this mailing list. 
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Each e-mail message can be traced back and forth from TOC WWW page shown in Figure 2. We have set 
two conditions; one was in which summary sentences were given, and the other was in which it was not 
given. 

4.1.2 Subjects 

Subjects were 20 undergraduate students forming ten pairs. The reason for making group of two is that the 
subjects could discuss naturally with each other, and therefore, we could collect natural speech proto-cols 
easily from the conversation [13] [14]. They were randomly assigned to two different experimental settings 
as described previously in this paper. 

4.1.3 Procedure of the experiment 

The two subjects were seated in front of the computer and were given an instruction for the present ex- 
periment by researcher. After the e-mail reading, the subjects were asked to answer some questions on 
reading comprehension. The subjects were allowed to start reading e-mail messages from anywhere. Then, 
they read e-mail messages through WWW pages for 20 minutes. After that, they were asked to answer some 
questions regarding particular content in the e-mail messages within 15 minutes. Finally, they were 
interviewed about the provision of advance information of the mailing list and the interest on the topic of 
discussion. None of the subjects know about this mailing list. 

4.2. Analysis and Results 

In this experiment, we recorded peer protocol with a digital video (DV). Then, we played the recordings and 
type in the conversation by listening the recordings. During the analysis of utterance, we identify sev-eral 
reading strategies or procedure for sentence comprehension. For each unit of procedure and strategy, the 
protocol was classified into the protocol categories [13]. For the classification, we have used the result of the 
factor analysis as we seen in Table 1. The categories “Read the content in detail and memorize” and “Use 
Table of Contents” were found to be significantly different on factor analysis. In the first cate-gory, we have 
considered utterances if the subjects read particular personal name and place name aloud. In the second 
category, we have considered utterances if the subjects read aloud the summary sentences or pursue 
continuity of the mailing list by pointing to the TOC. Some subjects pointed using mouse cur-sor’s move or 
their fingers. 



Table 3: Frequency of strategies appearance between two categories 



Protocol category 


With summary 


Without summary 


1 . Read the content in detail and memorize 


128 


103 


2. Use Table of Contents 


35 


22 



Table 3 shows the comparative result with two protocol categories. As the number of subjects is very less, a 
clear conclusion could not be drawn. But as in Table 3, the frequency of category 1 was relatively higher 
than that of category 2. As a the result, by means of summary display tool with e-mail messages has been 
suggested as a method to manage a lot of reading strategies easily. However, though the differ-ence of the 
frequency does not contradict the results of ANOVA in the previous section, it does not show a significant 
difference in the comparison of ratios ( _ o.67,p > .10 )• 

In this experiment, a significant difference was not seen in the frequency of the e-mail reading strate-gies. 
We need to add the number of experiments as well as study the influence of experimental design in peer 
conversation. 



5 Conclusions 

The results of this research may be summarized as follows: 

1. We applied the summary extraction method for mailing list, and analyzed e-mail reading comprehen-sion 
and reading strategies for reference. Although the result is limited to the e-mail messages we used, the 
display of e-mail summary sentences affects experimental subjects’ reading strategies. On the other hand, 



o 

ERIC 



38 



the result of comprehension test does not show significant differences. We may con-clude at this point that 
the method of summary sentence extraction is effective in understandings of relationship of e-mail 
messages. 

2. The influence of summary display on the e-mail reading strategies was examined from the analysis of 
utterance protocol. The use of Table of Contents WWW page along with e-mail summary sentences does 
not make a difference in the frequency of utterances, but preferential trend for the use of e-mail summary 
sentences was observed. 

As a problem yet to be solved in the future, we are interested in examining the effectiveness of reading 
strategies when e-mail messages are posted and read in real time. 
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In this paper, we describe our experiences in designing and using a multimedia 
composition-making system for children. The system allows children to make 
compositions using pictures, sounds and text. Moreover, it also allows pictures 
in the composition to be animated. We experimented with children using this 
system in three d ifferent settings. In the first setting, no topic was assigned to the 
children. In the second and third experiments, children were given a topic 
(different for each experiment) for composition related to their activity. We 
present here the results of our experiments and comment on how the constraints 
imposed by the topic affect children’ s expressive abilities. 

Keywords: animation, children’s expressive abilities, constraints and 
creativity, multimedia composition. 



1 Introduction 

In recent years, many Esearchers have studied multimedia techniques and have incorporated them into 
various educational systems. For example, Silva [5] described a multimedia soundscape system, “They Are 
Catching Sounds in the Park!”, for environmental education. In this system, children search for sounds by 
clicking anywhere in the picture. When they click an appropriate object, its associated sound and 
information are presented to the children. Bma [1] proposed a system for composing and writing stories via 
cartoons. Harviainen [2] presented a co-authoring system in which many users work together to compose a 
story. Ishii [3] and Kawakami [4] have developed other systems for making stories with multimedia. All this 
research demonstrates that multimedia has much potential for stimulating the ability of children to express 
themselves. In particular, we find that children can express their creative and imagined ideas much better 
with pictures and words than with words alone. Moreover, if we add an ability to attach sounds to pictures, 
and allow pictures to be animated, then this expressive power increases considerably. 

Motivated by these factors, we have developed a system to help children write multimedia compositions, 
and have tested it with children in three different settings. In this paper, we describe our system and report 
on our experiences with children using the system. 



2 Prototype of the System 

We developed a prototype of a multimedia composition-making system. Using our prototype, 

Children can express their thoughts and ideas via pictures, sounds, text, and animation sequences. In 
our system, children must first choose a background scene, in which they can then insert picture 
objects, sounds, and text. 

- Except for the background, children can attach sounds and text to picture objects, and can animate 
them to make a multimedia composition. 

This system has two modes: a ‘ Set up’ mode for the teacher or supervisor to allow them to determine which 
background scenes, picture objects, sounds, etc. are made available to the children for writing a composition, 
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and a ‘ User’ mode for children to write compositions. 



The ‘ Set up’ mode has the following two functions: 

1) Select situation: Set the context or theme for the composition. 

2) Edit situation: Set the categories of background scenes, picture objects and sounds corresponding to a 
theme. 



The * User’ mode has the following seven functions: 

1) Select background scenes. 

2) Select picture objects. 

3) Select sounds. 

4) Write text. 

5) Animate composition. 

6) Save composition. 

7) Load composition. 




Animate 



Insert Picture 



Attach Sounds 



Attach Text 



Load 



End 



Figure 1 . An example of the main window 

By double clicking on a picture in the main window, the sound attached to that object (if any) can be heard. 
Also, when the picture of an object is selected in the main window, the text attached to it is displayed in the 
text box. 

The animation module has five functions: show picture, hide picture, output sound, show text, and move 
picture. 

To replay animation, children click the ‘start’ button in the animation window. When the button is clicked, 
the system starts the animation sequence as previously specified. It replays each action one by one, but it 
pauses when the action is ‘ show text’ . To continue from there, the user needs to click the ‘ start’ button again. 



3 Experiments with the system 

We did three different experiments in which 
children used our system. In each experiment, the 
setting and the tasks given to children were 
different, as described below. 

3.1 Experiment I 

In this experiment, we studied a constraint-free use 
of the composition system. The children were not 
given any specific topic of composition, and they 
could use the system any way they like to create 
any composition freely. We prepared 54 
background scenes, 185 pictures and 68 sounds. 
Because no topic was given, children chose a, 
variety of themes. 




Figure 2. Window for the replay of animation 



3.2 Experiment II 



In this experiment, we introduced a constraint by giving a topic of composition to the children, and analyzed 
the generated compositions. The experiment was done at an activity center for children. At this center, 
children of each grade come periodically, and play or make some handicraft. One of the handicraft projects 
for third-grade children was making kites. So, the following week, we asked the children to make a 
composition about kite making. For the experiment, we prepared a version of the system with six 
backgrounds scenes of craft rooms. Three of these were scenes with kites in them, and the others were 
scenes with only a room and tables without kites. We also prepared 68 pictures and 35 sounds appropriate 
for kite-making activity. 

33 Experiment in 

In this experiment, we introduced a tighter constraint by 
giving a more specific topic of composition to children, and 
studied its effect. We asked the children to make a 
composition for the story “The coward king and robbef* 

(original title in Japanese). The original story is written in 
Japanese. At the same activity center for children used in 
Experiment II, the children made an extended version of this 
story, made a picture book to illustrate various scenes in the 
story, and then told the story using these pictures at their 
Christmas party. The week following the party, we asked the 
children to make a composition for this story using our 
system. 

For this experiment, we prepared a version of the system 
with eight background picture scenes related to the story. 

We also prepared 66 picture objects and 33 sounds 
appropriate to the story. 

In this experiment, we were interested in analyzing the 
differences between compositions made using our system 
and the corresponding pictures in the picture book for this story that the children had made earlier. We used 
the following method for computing the difference. The picture objects were grouped into ten categories, 
and the difference between two pictures (with the same background scene) was calculated as follows: 

For each picture object category: if there is an object of that category in both the pictures, we 
say that the difference between the two pictures with respect to that category is zero. If one 
picture has an object from that category, and the other has none, we say that the difference with 
respect to that category is one. The difference between two pictures is the sum of the 
differences over all ten categories. 

Figure 3 shows the result of applying this procedure. 

We see that the differences for the climactic scenes (scenes 6* ?) are higher than the other scenes. 

3.4 Discussion 

An analysis of the compositions produced in the three experiments is shown in Tables 1, 2 and 3. Table 1 
shows the average number of compositions produced by a participant in each experiment. We see from it 
that the children were most expressive when the topic was most constrained (Experiment III). 



Table 1. Number of compositions per participant 





Experiment I 


Experiment II 


Experiment III 


Number of participants 


11 


7 


5 


Number of compositions per 
participant 


1.3 


1.4 


4.6 




Figure 3. Differences between the 
composition picture and the original 
picture from the picture book (for each 
background scene) 



Table 2 shows a more detailed analysis of compositions with respect to how multimedia features of the 
system were used. 



42 



Table 2. Number of multimedia features per composition 



Multimedia feature 


Experiment I 


Experiment II 


Experiment ID 


Picture objects 


3.6 


11.2 


6.9 


Sound attachments 


1.1 


2.4 


1.6 


Text attachments 


- 


1.1 


1.6 


Animation 


1.0 


2.8 


9.9 



Here we see that picture and sound attachments are used most in Experiment II. This may be because in this 
setting children were describing a situation using generally one page (screen). For this, they used many 
objects and sound attachments to provide information about the depicted situation. We also see that text 
attachments and replay actions were used most in Experiment III. It might be because in this setting they 
were describing a story, for which text attachment is a useful way to express characters’ utterances, and 
animation is useful to express characters’ movement. We also would like to point out that in Experiment III 
there were fewer picture objects and sound attachments per composition. This is because to show the flow of 
events in the story, children made many compositions (Table 1). 



Table 3. Analysis of animation operations per composition (in percent) 



Animation operation 


Experiment I 


Experiment II 


Experiment III 


Show picture 


0 


0 


53 


Hide picture 


0 


0 


4 


Output Sound 


13 


0 


10 


Show text 


- 


0 


15 


Move picture 


87 


100 


18 



In this table we see that in Experiments I and II, mostly the ‘ move picture’ operation was used. In 
composing a story, however (Experiment III), the ‘show picture’ was used most. We also found that the 
* hide picture ’ operation was rarely used. 



4 Conclusions 

From our experiments, we see that the multimedia features of the composition-making system are most 
useful in illustrating a story or a narrative. Sound and text attachments and animation operations can be very 
helpful in expressing movement of characters and the progression of events in a narrative. We also found 
that many children are most expressive when they are given a focus of composition. 

From these results, we propose that a system such as ours can be used in the classroom for children to make 
compositions about field trips and class excursions. For each trip or excursion, the teacher can set up the 
system appropriately by choosing relevant picture and sound libraries before children use the system. In this 
way, we feel that our system can provide a step forward from Silva [5]. Children are more actively involved 
in making compositions with our system than in exploring with ‘ They are catching sounds in the park!” 
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In this paper, we propose a technique for reducing processing time during program 
evaluation, and examine processing time of evaluating programs of which sizes are 
relatively large in novice programming courses. We proposed a method of 
constructing an automated evaluation system assisting teachers teaching novice 
programming. Our system evaluates learners’ programs by comparing them with a 
standard algorithm representing teacher’s intentions. By using our system, teachers 
can easily pick up learners’ defective programs. We constructed a prototype system, 
and examined whether the system can evaluate programs actually submitted by 
learners. We confirmed that it could evaluate the programs validly. However we 
found that we should improve the processing time after evaluating various programs. 
In order to reduce processing time, we extend the matching algorithm using two 
ways. As a result, processing time is improved without spoiling the accuracy of 
matching. After that, we design a model course of novice programming based on 
actual courses in our university. And we examine the relation among program size, 
arbitrariness of teacher’s intention and processing time. Then we confirm that the 
processing speed of our system is fast enough to be used in actual education 
environment. 

Keywords: educational system assisting teachers, automated evaluation system, 
program diagnosis, experimental evaluation. 



1. Introduction 

By using program diagnosis technique, many programming education systems have been developed[l][2]. Most 
of them are designed to help learners, not the teachers. We think it is necessary to help teachers in order to give 
learners better advice. It needs much effort for teachers to evaluate many programs. So we constructed a 
prototype system assisting teachers teaching novice programming[4]. We have designed a model course on the 
basis of actual novice programming education course. Then, we have examined whether the system can evaluate 
programs written by learners. We confirmed that it could evaluate the programs validly [5]. However we found 
that we should improve the processing time after evaluating various programs. In this paper, we propose a 
technique for reducing the processing time (section 3), and examine processing time of evaluating programs of 
which sizes are relatively large in novice programming courses, then confirm that the processing speed of our 
system is fast enough to be used in actual education environment (section 4). 



2. Our previous work 

Generally, teachers teaching novice programming arrange goals for their exercises, and set exercises related to 
the goals. We call these goals “ teachers intentions”. They evaluate whether each program submitted by a learner 
achieves the goals or not, and advise the learners according to the result of the evaluation. However, it needs 
much effort for them to evaluate many programs with various bugs. So we support them by developing an 
automated evaluation system classifying programs which perfectly satisfy their intentions , which partially satisfy 
their intentions or do extra work, and which don’t satisfy their intentions. Therefore they only have to check the 
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unsatisfied programs carefully. 

We found that most of teacher s intentions can be represented with standard algorithms to solve exercises. So 
we use the standard algorithms for inputting teacher £ intentions. We designed an algorithm representation based 
on PAD expression. We call the representation “Extended PAD”. By using the Extended PAD, teachers are 
allowed to use the following structures in order to represent arbitrariness included in their intentions. 
Non-ordering structure: It represents arbitrariness on the order among tasks, 

Alternative structure: It represents arbitrariness on method to achieve a goal. 

The Extended PAD consists of two types of elements: elements which correspond to a Pascal operation or 
control structure, and which correspond to a sequence of Pascal operations. We call the previous elements 
“primitive operations”, and the other “macro operations”. 

We developed a method matching learners’ programs with a standard algorithm. According to this method, 
the system tries to check correspondence of every combination of statements in a learner’s program with the 
statements in a standard algorithm. If it judges that a statement in a learner’s program matches with a statement 
in a standard algorithm, it makes a hypothesis on a correspondence of these statements and correspondences of 
variables referred by the statements. Then it continues matching the other statements on the basis of the 
hypothesis. As the matching process succeeds, the hypothesis grows up. Generally, possible correspondence of 
variables is not unique. So the other hypotheses containing the other correspondences of variables are set up at 
the same time. According to the result of matching, it outputs its judgement, “perfect match”, “partial match” or 
“no match” based on the most plausible hypothesis. “Partial match” means that a learner’s program doesn’t 
match with a standard algorithm perfectly, but both the ratio of matched statements to the whole in the learner’s 
program and the ratio of matched statements to the whole in the standard algorithm are higher than each 
threshold we defined. 



3. Improvement of processing time 

In order to reduce processing time, we should re-consider a method of matching programs. The system outputs 
the result of judgement on the basis of the most plausible hypothesis. The others are rejected. So if the system 
has an ability to find useless hypotheses on the way during matching process and avoid checking the 
correspondence of statements on the basis of the useless hypotheses, processing time must decrease. 

So we extend the matching method as follows: 

1) When the system intends to make new hypothesis containing correspondence of a new combination of 
statements in a learner’s program and a standard algorithm, it calculates the ratio of matched statements to the 
whole under the assumption that all of the following statements will be matched perfectly. If the ratio cannot 
reach the threshold, the system doesn’t make the hypothesis and omits the process of matching the following 
statements based on the hypothesis. 

2) After the system has matched whole statements on the basis of a certain hypothesis, it tries to match 
statements on the basis of another hypothesis. In such a case, when the system finds that the ratio of matched 
statements to the whole cannot reach the ratio of previous trial, under the similar assumption to 1), it stops 
matching the following statements. 

As a result, processing time is improved without spoiling the accuracy of matching. We confirm that 
evaluation of programs is not changed by the extension. Table 1 shows improvement of processing time 
necessary to judge the programs. Exercise (1), (2) and (3) are illustrated in Table 2. The computer system used 
for both experiments before and after the extension is an Engineering Workstation JU 2/2300 (CPU: Ultra 
SPARC-II (300MHz) * 2, SPECint95: 12.3, SPECfp95: 20.2, Operating System: SunOS5.6, Made by: Japan 
Computer Corp.). 



Table 1 : Improvement of processing time. 





Number of programs 


Average time of processing [sec/program] 


Before extension 


After extension 


Exercise(l) 


42 


184.98 


1.57 


Exercise(2) 


56 


1.43 


0.13 


Exercise(3) 


49 


109.06 


1.10 



4. Applicability of our system to actual education environment 

In order to discuss applicability of our system to actual scene in education, we design a model novice 
programming course based on actual courses in our university[5]. The exercises in the course are seen in Table 2. 
We use programs submitted by learners in the actual courses of our university. 

We write each standard algorithm of exercise within the following restriction: the number of steps of 



46 



Extended PAD must be less than twice the number of steps of a standard program for corresponding exercise. 
The reason is that teachers don’t prefer writing more detailed standard algorithms because of their costs. 

The computer system used for this experiment is also the Engineering Workstation JU2/2300. 



Table 2: The exercises in the model course. 



Exercise(l) 


There are several birds and tortoises. Find the number of them when the following conditions are given: 
(1) Sum of the heads of birds and tortoises. (2) Sum of the legs of birds and tortoises. 


Exercise(2) 


A character datum that is an uppercase character will be input by user. Convert it to lowercase. 


Exercise(3) 


A hexadecimal number will be input by user as two digits of character datum “0-9” or “A-F\ Convert it to a decimal number. 


Exercise(4) 


Solve an equation “ax A 3+bx A 2+cx+d=0” by using the Newton method. 


Exercise(5) 


Sort integer data in an array by using the straight selection sort algorithm. 


Exercise(6) 


Solve simultaneous equations by using the Gaussian elimination. 



4.1 Result of experiment 



In order to evaluate processing time, we must consider the following factors: program size of standard 
algorithms and one of learners’ programs, arbitrariness of teachers intentions , judgement of learners’ programs. 
Therefore we examine the relation among these factors and processing time. Table 3 shows: 

- Program size of a standard algorithm and arbitrariness of teacher s intention. 

- The number of learners’ programs and average of their program size. 

- Judgement of learners’ programs, and the average processing time. 

We measure the program size by the number of statements, blocks and variables, and the arbitrariness by the 
number of alternative structures and non-ordering structures including a standard algorithm. The judgement 
means the number of learners’ programs that are judged “perfect match”, “partial match” and “no match” by our 
system. 

As program size increases, the number of combination of statements in a learner’s program with the 
statements in a standard algorithm also increases. At the worst case, the number of the combination increases 
proportionally to factorial of number of the statements. However Table 3 shows gentler increasing. From the 
result, we think our extended matching method works well. Exercise (4) needs rather long processing time. We 
think that the number of statement per block is larger than the other exercises, and most of statements in the 
block have dependencies, i.e. a variable assigned some value by an assignment statement is often referred in the 
following statements in the same block. In such a case, the system must compound formulas and the process of 
compound needs rather long time. However, we check how often such an exercise appears in actual textbook[3] 
for novice programming and find that only one exercise. 



Table 3: Relation among program size, arbitrariness of standard algorithm and processing time. 





Standard algorithm 


Learners’ programs 


Judgement of 
Learners’ programs 


Processing 

time 

[sec/program] 




Number of: 


Averaj 


;e of: 


Number of: 


State- 

ments 


Blocks 


Vari- 

ables 


Alter- 

native 


Non- 

ordering 


Pro- 

grams 


State- 

ments 


Blocks 


Vari- 

ables 


Perfect 

match 


Partial 

match 


No 

match 


(i) 


5 


1 


4 


1 o 


0 


42 


9.2 


9.1 


1.4 


0 


4 


38 


0.96 




!?! 


4 


1 


2 


0 


0 


56 


9.3 


2.1 


3.1 


3 


17 


36 


0.30 




(3) 


11 


5 


4 


0 


0 


49 


19.7 


7.7 


5.9 


0 


0 


49 


1.50 




w 


9 


4 


m 


^ o: 


0 


14 


9.9 


2.0 


6.6 


7 


1 


6 


6.23 




(5) 


10 


4 


5 


0 


0 


14 


9.8 


4.0 


5.1 


1 


10 


3 


0.30 


(*X#) 




10 


4 


5 


i 


0 


14 


9.8 


4.0 


5.1 


1 


10 


3 


0.31 


(*) 




10 


4 


5 


2 


0 


14 


9.8 


4.0 


5.1 


1 


10 


3 


0.43 


(*) 




10 


4 


5 


3 


0 


14 


9.8 


4.0 


5.1 


1 


10 


3 


0.44 


(*) 




10 


4 


5 


0 


1 


14 


9.8 


4.0 


5.1 


1 


10 


3 


0.35 


(#) 




10 


4 


5 


0 


2 


14 


9.8 


4.0 


5.1 


2 


9 


3 


0.52 


w 


(6) 


28 


12 


11 


0 


0 


24 


27.3 


11.8 


11.0 


1 


11 


12 


9.87 


<*) 




28 


12 


11 


11 


0 


24 


27.3 


11.8 


11.0 


1 


11 


12 


31.00 


(*) 



We evaluate our system on another standpoint. As arbitrariness of teacher s intention increases, possible 
combinations of statements in a learner’s program with the statements in a standard algorithm also increase. We 
prepare some standard algorithms for the exercise (5) and (6), and process actual programs of learners. The result 
is seen in the lines with (*) and (#) in Table 3. 

We prepare a standard algorithm including three alternative structures representing two methods to achieve a 
goal. At the worst case, the number of combination of statements in a learner’s program with the statements in 
the standard algorithm increases 8 (=2 A 3) times. However Table 3 shows 1.5 times increasing. We also measure 
processing time by using a standard algorithm of exercise (6) including 1 1 alternative structures representing two 
methods. Although it seems that processing time increases 2048 (=2 A 11) times, Table 3 shows only 3 times 
increasing. Similarly, on exercise (5), we prepare a sample of Extended PAD including 2 non-ordering structures 
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representing arbitrariness on the order between two tasks. In this case, it seems that processing time increases 4 
(=(2!) A 2) times. However Table 3 shows 1.8 times increasing. We think that the reason why the processing time 
isn’t so increased is that our extended matching method works well. When our system evaluates using standard 
algorithms that include alternative structures or non-ordering structures, useless hypotheses are rejected at earlier 
stage of matching process. 

In the next section, considering the result shown in Table 3, we discuss applicability of our matching method 
to actual education. 



4.2 Estimation of processing time for actual exercises in novice programming 

We survey model programs in a textbook[3], and write standard algorithms considering alternative coding 
methods or alternative order of statements. Then we investigate the program size, numbers of alternative 
structures and numbers of non-ordering structures. The result is seen in Table 4. 

We can regard that our matching algorithm processes even programs belonging to the most complicated class 
in actual education within practical processing time. The reasons are as follows: 

- The number of statement in programs of high level is less than the exercise (6) on average. And the largest 
number of statement is equivalent to the exercise (6). 

- The number of alternative structure is almost 4 for programs of middle or high level. 

- The number of non-ordering structure is less than 2. 

- Therefore we think that our system can evaluate the most complicated program in the textbook with 
processing time similar to exercise (6) (31.00s/program). 

- In fact, if a teacher takes care of 100 students in a class, the time necessary to evaluate their programs by our 
system is only 50 minutes or so. We think this is practical enough. 



Table 4: Program size of exercises in a standard textbook of novice programming. 



Exercise 


Number of 


Number of 


Alternative structures 


Non-ordering 


programs 


statements 


Iteration 


Others 


structures 


Basic level 


(the four rules of arithmetic, etc.) 


19 


8.5 


1.1 


0.2 


1.2 


Middle level 


(sorting, etc.) 


34 


17.9 


4.0 


0.5 


1.1 


High level 


(searching trees, etc.) 


14 


20.9 


3.5 


0.3 


1.1 



5. Conclusions 

In this paper, we extend our matching algorithm to improve the processing time. Through the examinations and 
the discussion about processing time, we confirm that our system can evaluate actual programs in novice 
programming in practical time. Now, we are constructing a graphical user interface for describing standard 
algorithm. We will evaluate usability of our system after constructing it. 
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This paper is a study on the design and implementation of the cooperative 
monitoring system using a mobile agent for an educational portal site. Generally 
educational portal sites have many addresses of teacher's homepage related 
education. Therefore, portal site has a very difficult task with maintaining a 
consistent address of site as well as it is impossible that administration of portal 
examines all dead sites in searching education site and DB. In order to solve this 
problem, we designed and implemented a mutual cooperative monitoring system 
to filter off dead site using a mobile agent. This monitoring system applies to the 
Korean educational portal site (KEPS) for elementary students and teachers. For 
efficiency this system, we made an experiment that compared a cooperative 
monitoring agent system with a stationary monitoring agent system. 

Keyword: Education Portal Site, Cooperative Monitoring System, And 
Mobile Agent 

1 Introduction 

Today, the advent of the web that can easily be connected through the "Internet" is known to be an easy and 
popular method for teaching and learning. Web-based educational homepages are used in many computer 
assistance medias and also the numbers of educational sites are on the increase extremely. 

An extremely increase in number of homepage raises a question whether a student can search appropriate 
homepage for learning. In case of finding educational contents using a general searching engines, the 
searched site can exist an irrelevant contents against a student’s request. Moreover the result of searching 
content fell into learning confusion, because the contents are difficult to apply at learning intact. 

In order to overcome this problem, an educational portal site was constructed to gather only educational 
homepages that had been made several times before. An advantage of educational portal site is that content 
is used correctly and rapidly in learning because searching site is well constructed. In addition student can 
easily get suitable contents. For gathering of an educational homepage, an educational portal system, called 
KEPS, was constructed by the EDUNET and Inchon National University of Education. 

While walking past a type of the gathered homepage in KEPS, it can be seen as to make not by an expert 
institution or a special company but by a teacher and a private person. As a result, characteristic of the 
homepages have to be petty and is frequently updated. Because the educational homepage can disappear 
easily, portal site faces difficulty to maintain consistency of the site address. If a hyperlinked address of a 
portal site is not connected or the retrieval site is disappeared to user, then this portal site may bring discredit 
to student. In order to maintain consistency of portal, the administrator of portal site must validate all 
addresses of site. But this examination is impossible work that man completely manages and finds. 
Consequently, a monitoring of a site address for finding the dead site can be process by an intelligent agent 
instead of human. 

A single agent needs comprehensive amount of time required for the monitoring of a portal site. If a single 
agent examines extremely a many site addresses, the monitoring work may be inefficient. Because a mobile 
agent is possible with decentralization and a parallel processing, the monitoring works using a mobile agent 
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can be process effectively [5]. 

Accordingly, this study designed and implemented a mutual cooperative monitoring system to filter off dead 
site using a mobile agent. In the following section, the mobile agent and monitoring scheme will be surveyed 
and the overview of the structure of monitoring agent will be designed. And the next section will be focused 
on implementation and experimentation of monitoring agent system. Finally the conclusion and future works 
will be described. * 

2 Mobile Agent and Cooperative Monitoring 

The agent is a program with intelligent characteristics to help the users with the use of computers and take 
the user's place. The intelligent agent perceives any dynamic stimulation or condition and interprets the 
data collected for a solution to the problem and exercises reasoning for a final decision. It also acts to 
change the conditions within its environment in order to perform assigned duties. It has autonomy, social 
ability, reactivity, pro-activeness and a cooperative relationship, learning, mobility, and so on [9]. 

Generally an agent divides a kind of two by the mobility, a stationary agent to be executed roles in single 
system, while the mobile agent is executed at various systems after moving through the networks. An 
execution example of the mobile agent is shown in figure 1 and the mobile agent based environment is 
viewed figure 2. The mobile agent server must be installed to act a mobile agent as figure 2. 
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Figure 1. Execution of mobile agent 



Figure 2. Mobile agent based environment 



The mobile agent has a specific characters listed below compared with a stationary agent [5] [6]. 

• The mobile agent reduces the network load. 

• The mobile agent overcomes network latency. 

• The mobile agent encapsulates protocols. 

• The mobile agent executes asynchronously and autonomously. 

• The mobile agent adapts dynamically. 

• The mobile agent is naturally heterogeneous. 

• The mobile agent is robust and fault -tolerant. 



In the information retrieval, a monitoring work ascertains a state of gathering sites for the maintenance of 
data consistency. Generally, because the information of the web is changed frequently, a monitoring job by 
human is an impossible or inefficient work. This monitoring job can be processed by intelligent a computer 
program instead of a human. Such a program is called the web robot or an intelligent agent system [ 1 0][ 1 1]. 

In case of examining many sites in the monitoring work, if a single agent of the only server processes 
monitoring work, then the monitoring work may be needed long time and overloading of a monitoring server. 
The mobile agent has made possible cooperative and speedy monitoring job from distribution and parallel 
processing [8][1 1]. 



3 Cooperative Monitoring System 



3.1 Overview of System 
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Overview of the KEPS system, including the temporary monitoring agent system is shown figure 3. 




The portal system is consisted of four parts. There are the portal web server (PWS) and the monitoring agent 
server (MAS), the temporary monitoring server (TMS), a mediator. For using educational portal service, 
user must be connected with the Portal web server. Gathered address of an educational homepage is 
supported searching service of an education contents to user through the Portal web server. The Portal web 
server has searching engine, site DB and a query processor. The monitoring agent server has a stationary 
monitoring agent and a cooperative mobile agent, error DB, a mobile agent server. Also the monitoring 
agent server performs works as a creation and an allocation, a control, a gathering of the monitoring mobile 
agent. For the mobile agent perform it’s task fully, each server is installed the mobile agent server 
necessarily. 

The temporary monitoring servers are in existence out the KEPS system. In order to process a fast 
monitoring work, the TMS have function of distributed and parallel processing. The number of TMS is not 
fixed but dynamic by amount of monitoring job. Furthermore the TMS is used in temporary palace which 
mobile agent examines each a state of the registered site. At ordinary times, the TMS is not used usually for 
examining a state of the registered site. However the TMS can be only used when is requested by the 
mediator agent server. 

The mediator is situated between the monitoring agent server and TMS, and acts as the role of mediation 
with the mobile agent and servers. All agents and agent servers must be registered in the mediator. 

3.2 Design of KEPS System and Cooperative Work 

The structure of the KEPS System is detail shown figure 4. The portal web server is consisted of searching 
engine and query processor, is shared the gathering DB of portal site. The searching engine provides 
searching service about education content and the query processor is shown the result searching at DB. The 
monitoring agent server is consisted of inference engine and agent manager, error DB. The monitoring 
system in monitoring agent server has a stationary agent and a mobile agent for distribution and parallel 
working. A stationary agent examines the state of gathering site and the confirmation of HTML documents 
through HTTP connection. After a failure sites are saved at temporary error DB, these will be deleted from 
site DB of portal web server. A permanent deletion of fail sites is executed by inference engine of the 
monitoring agent server. 

When a monitoring agent server is overloaded or the stationary monitoring agent has difficulty processed by 
examination with many site, the monitoring agent server requests to the mediator about information of the 
registered TMS. If the number of the TMS is lacking, the monitoring agent server waits until the TMS 
becomes sufficient. Having sufficient number of the TMS, the mobile agent is created to divide as a suitable 
size of address by inference engine. And then the mobile agent has been created by a monitoring agent 
server, will be cloned with suitable number. Each mobile agent is allocated a monitoring work and will be 
dispatched to the TMS through ATP connection. The mediator agent can grasp each work states of an agent 
by using the agent finder. 
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Each agent is moved to temporary monitoring server and examines the allocated addresses of sites through 
HTTP. When a mobile agent is finished all checking of sites, it sends to the monitoring agent server with the 
result of observation. If the job of the mobile agent is occurred some problem, monitoring agent server 
creates a new mobile agent and re -dispatches to the TMS. All results gathers, result of examination saves at 
site DB and error DB. Finally, dispatching the agents retracted by the monitoring agent. 




The processing algorithm of execution about monitoring working is shown figure 5. The job of monitoring 
using the mobile agent has advantages that prevent an overloading of a single server and lessen monitoring 
time by distribution and parallel processing. Because agents are not used stationary server but are 
dynamically used in other servers, all servers performed share resources of monitoring system. Accordingly, 
each agent can do cooperative parallel processing using autonomous and society properties of agent. 



4 Implementation and Experiment 

4.1 Implementation and Application of System 

The monitoring agent system proposed in this study was implemented two types. The stationary monitoring 
agent was implemented by using VC++ and CLIPS. Also the mobile monitoring agent system proposed in 
this study was implemented using JAVA based Aglet API and JESS. Aglet is the java class library for that 
can easily design and implement all the properties of the mobile agent. Moreover the Aglet provides with the 
Tahiti server and Agent finder for helping research of users. 

The stationary monitoring agent interacts with the mobile agent of Tahiti server based environment. 
Inference engine of the stationary monitoring agent was used the CLIPS dynamic linked library and the 
mobile monitoring agent system was used the JESS class library. The CLIPS and JESS are rule based 
inference engine and was used to infer planning and allocation of the mobile agent. SQL was used for the 
gathering DB of portal site. ODBC and JDBC were used to connect the monitoring agent system and the 
gathering DB of site. 




Figure 5. Algorithm of monitoring procedure 



Figure 6 below is image of the interface of the stationary monitoring agent by making VC++. Figure is 
shown that the single monitoring agent is examining each site. The stationary monitoring agent was 
consisted of three parts mainly. The left screen of figure is represented list that the agent will examine site of 
DB. Also the center of screen is viewed results of a successful site and the right screen is represented results 
of a failure site. 




Figure 7 is shown screen that the mobile monitoring agent is examining each site with distribution and 
parallel processing. If the numbers of sites are many in existence, the stationary monitoring agent executes 
the mobile agents to interact with the Tahiti server as followed image. Above window of figure is 
represented the stationary monitoring agent. Black screen below is viewed that mobile agent sever is 
executed by the stationary monitoring agent. Small screen below is shown the Aglet viewer. The Aglet 
viewer perform an important role as a creation, dialog, dispose, cloning, dispatching, retracting of a mobile 
agent. 
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Figure 7. Interaction between stationary monitoring agent and Tahiti server 



In order to use the implemented monitoring system in this study, we applied at the educational portal system 
and the KEPS system in the EDUNET server. Figure 8 is shown the searching screen of the web browser 
using KEPS system. This portal site in the EDUNET was constructed for the Korean elementary student and 
teacher. Also this site contains all contents about the curriculum of the Korean elementary school. 
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Figure 8. Searching of KEPS system 



4.2 Experimental Results 

For examining the efficiency of the cooperative monitoring system using the mobile agent, we compared and 
evaluated a monitoring time of each agent system. A comparative and estimative items listed below are as 
followed. 

• Comparative item 

- The single stationary monitoring agent vs. the cooperative monitoring agents. 

• Estimative items 

- The monitoring time of the single monitoring agent 

- The monitoring time of the cooperative monitoring agents(3) 

- The monitoring time of the cooperative monitoring agents(7) 

- The number of sites: 10,30,50,70,90,110,130,1 50,170,190 ..etc. 
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The experiment measures examination time of sites using a comparative and estimative items above. The 
estimative result is shown Table 1 and is represented figure 9 with form of graph. The horizontal axis of 
graph is represented the number of site and the vertical axis of graph is represented monitoring time of each 
agent. 

In case of the number of an examine site is small, the result of experiment is viewed that the single 
stationary agent is faster speed of examination than the mobile monitoring agent. Also, when mobile agent is 
dispatched to three servers, speed of examination is faster than is dispatched to seven servers. The reason is 
caused by overtime occurred because the many mobile agents are created, allocated, gathered. 

However, the more the number of site increases, the faster the mobile monitoring agent gets speed of 
checking than the single stationary agent. In particular, when the cooperative monitoring system using many 
agents, experimental result is shown that a speed of examination is very fast. If a single stationary agent 
processes very many sites, the result of execution can be useless though the result is very accurate. 

Consequently, the cooperative monitoring agent can become higher execution speed by distributed and 
parallel processing and an overload of network by using a mobile agent can be decreased. If a server has an 
active environment of the mobile agent, the servers can be used with an active space of a searching agent 
and a monitoring agent. 



Table 1 . Result of monitoring time 



— -^-^^Numlbeir of Agent 
Agent Type "* 


10 


30 


so 


70 


90 


210 


130 


150 


170 


190 


Stationary Monitoring Agent 


42 


137 


201 


261 


374 


412 


518 


592 


645 


743 


Mobile Monitoring Agent(3 Servers) 


120 


132 


143 


165 


221 


253 


262 


282 


316 


335 


Mobile Monitoring Agent(7 Servers) 


130 


121 


124 


122 


148 


143 


147 


183 


186 


182 



Time(sec) 




10 30 50 70 90 110 130 150 170 190 



Figure 9. Graph of monitoring result 



5 Conclusion and Future works 

This study is on the efficiency of cooperative monitoring agent using mobile agent for educational portal site. 
The monitoring job has been getting difficulty processed by human. Thus, an intelligent agent can process 
the monitoring of the portal site instead of human. A monitoring work by using a single stationary agent 
needs long time for checking of many sites. 

In order to overcome the problem in this study, the mobile agent is used in monitoring job. The monitoring 
job of educational portal site can be processed by collaborative method of decentralization and parallel using 
the mobile agent. The monitoring system was implemented by using the Aglet and Tahiti server. This system 
could execute cooperative monitoring job through an intelligent interaction between the stationary agent and 
a mobile agent. Also the KEPS system is possible with the mediation and the registration of agents by using 
the mediator agent between the monitoring server and the temporary agent sever. 
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The temporary agent sever is not fixed with the number but can be dynamically changed. Therefore all 
servers are by resources of monitoring job and each server can exe cute its role by inference. 

More studies are required on research that constructs knowledge base for inference engine of the mobile 
agent. For effective portal site constructed, future work needs researches about not only intelligent 
monitoring but also htelligent searching and gathering of educational information. In order to interact 
between the mobile agents, we require research about KQML, language for sharing and exchange of 
knowledge between agent and agent. 
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This paper describes an analysis of the effectiveness of an in-service training 
system developed by a project sponsored by the foundation of the Information- 
technology Promotion Agency, Japan (IPA). We developed and carried out a 10 
days training course for 65 teachers in three different locations. The three main 
elements of this course were (1) training curriculum, (2) CD-ROM materials, 
and (3) Web-based support system. The participants of this course were hoping 
to become Information Technology (IT) leaders in their schools. An analysis 
was conducted to investigate the factors influencing the effectiveness of the 
training. Based on our analysis, we were able to draw the following conclusions: 

(1) Web-based training support system and CD-ROM materials were very 
effective to improve teachers’ knowledge and skills, regardless of prior 
knowledge and skills (2) Traditional instructional style (i.e. one-way instructor- 
centered style) was generally an ineffective training method.(3) CD-ROM 
materials significantly enhanced the effectiveness of teachers’ creating materials 
(such as a Web page), especially when the CD-ROM was used for self-study. 

These findings will be useful for educators and educational designers who plan 
and conduct in-service training programs. 

Keywords: Distance Education, Teacher Training System, Web-based 
Learning, Teacher Education, Training Program Assessment 



1 Introduction 

The Japanese Ministry of Education decided to introduce the Internet to all Japanese schools (i.e. elementary 
to high schools) by 2001 to promote students’ Information Communication Technology (ICT) literacy and 
the effective utilization of information tools to enhance subject-area learning. It has been reported that in 
order to realize information technology education, three major factors are crucial: Preparing sufficient 
hardware, providing excellent software, and promoting teacher’s ICT literacy (refs. [1]). Several papers 
describe certain barriers for integrating ICT into curricula (ref. [9], [5]). The Ministry of Education in Japan 
introduced several projects aimed at promoting teacher training by removing these barriers. One of these 
projects was named “Project for development and evaluation of in-service training system for improving the 
ICT literacy of schoolteachers” sponsored by the Information-technology Promotion Agency foundation 
(IPA). This project was divided into two sub-projects, one to develop ICT leadership of schoolteachers, and 
another promote the training of ICT coordinators. This article examines the development of ICT leadership 
of schoolteachers, especially focusing on the evaluation of in-service training system. 

The project consists of about forty members, joined by a variety of experts including engineers from 
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computer software companies, ICT-experienced teachers, and university researchers. The project has 
conducted over one year starting in 1998, and included the following elements: 

(1) Development of training curriculum and support system 

(2) Implementation of the training according to the curriculum using the support system 

(3) Analysis of effectiveness of the training system 



2 Outline of in-service training system 

The training system consists of major three components: (1) training curriculum, (2) CD-ROM materials and 
(3) Web-based support system. 

2.1 Training curriculum 

The guiding concept of the training curriculum is based on a constructivist approach (refs. [11]), and 
computer- mediated, problem-based training such as the learning from doing complex, challenging, and 
authentic problem through collaboration and communication (refs. [8]). The curriculum was designed to 
achieve the following goals: 

(1) To train ICT leaders to play the central role in promoting classroom teachers’ ICT literacy in each school. 

(2) To attach greater importance to practical skills (rather than computer operation skills) by using ICT tools 
in actual classes (refs. [5], [14]). 

(3) To improve basic skills of networking such as connecting to the Internet and configuring the school’s 
LAN. 

(4) To develop consultation skills for the leader to provide advise classroom teachers. 

(5) To provide a computer-mediated learning-support system for sharing knowledge dynamically. 

2.2 Development of CD-ROM materials 

CD-ROM materials were developed in cooperation with a computer software engineer, an expert teacher, 
and university researchers according to the training curriculum. Many resources related to the training 
course, including teachers’ case studies and video of actual classes using ICT tools which were recorded 
onto CD-ROM and used for instructor’s presentation and participant’s self-learning (refs. [6]). Figure 1 
shows an example of an interview scene of expert teachers on CD-ROM materials, five disks in total. 

2.3 Web-based support system 

Fig. 2 shows the configuration of a web-based support system for training. This support system consists of 
many modules such as database references, collaboration support, registration, and management of bulletin 
boards, checking items and participant’s response in order to realize sharing discussion, collaborative group- 
based works and participants’ reflection on each other’s work (refs. [3], [12]). Fig.3 shows an example of a 
bulletin board in the system where participants can submit and share their opinions. 



3 Design of system evaluation 

The training program was conducted in three different locations. Each program ran for a period of 10 days, 
and a total of 65 teachers participated. These participants were identified as candidates to become school 
ICT leaders. The program for the training course, which consists of 12 topics, is shown in table. 1. 

Evaluation plays a critical role in the improvement of a training system. Despite the importance of 
evaluation, little research exists to analyze the factors contributing to the effectiveness of the training system 
(refs. [7], [10], [13]). In order to obtain findings helpful to the design the training program, we carried out an 
evaluation according to the following scheme. Checking was done at pre- and post- training, and all training 
scenes (such as participants’ talk, activities, collaborative work and instructors’ presentations) were 
videotaped for measuring activities’ time and conducting qualitative analysis. 

We introduced a total 106 items organized into 12 topic areas. Participants were required to respond to these 
106 items before taking the course (pre-check), again just after taking the course (post- check), and one last 
time after the training was finished (after check). We compared differences among scores of pre-, post-, and 
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after check scores in the course. In order to analyze the factors influencing differences of pre- and post- 
scores, we carried out the regression analysis shown in a later section. 
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In the morning 


Afternoon 


Assignment 


Date 


9:00 10:30 10:40 


12:00 


13:10 14:40 14:50 


1620 


1630 1730 


i* 


Orientation 
Opening ceremony 


Present situation and 
some problems 
in ICT education 
Ministry of Education 
Lecture 


Direction of 
information society 

Lecture/Discussion 


Fundamental 
technology supporting 
ICT education 

Lecture& discussion 




T 3 


Contents and methods on ICT education 
Lecture and discussion 


Curriculum development of ICT education 
Presentation and group work 


Homework 
to develop 
school 
curriculum 


3 m 


To continue 
developing 
curriculum for 
each school 


Participants’ 
Presentation of each 
curriculum 
Discussion 


Technical Foundation on OS, Multimedia 
and network 

Demonstration and lecture 


Homework 
for topics 


4^ 


Foundation of network focusing on LAN in 
campus, design of configuration of LAN, 
and management of network system 

Demonstration and lecture 


Information ethics and 
privacy protection 
Lecture and 
discussion 


Producing 

homepage 

Demonstration and 
individual work 


Homework 

as 

homepage 

producing 


3 ”* 


Presentation of produced homepage 
Discussion How to make materials 


ICT utilization to subject learning and the role of 
computer and the Internet 
Lecture and discussion 


Homework 
for topics 


6 th 


How to develop a lesson plan and how to make materials 
using computer, hypertext and internet 
Presentation and group work 


Homework 
for materials 
production 


yih 


Continuing to materials production 
Group work 


Presentation of 
produced materials 
Discussion 


Collaborative learning 
Lecture/discussion 


Homework 

for 

checking 

consultation 


T 


How to develop consultation skill s 
concerning to ICT education 
Discussion and group work 


Practice of consultation 
Discussion and group work 


Homework 
for practice 


“9® 


How to promote and manage 
in-service training at each school 
Presentation and discussion 


Practice to make training plan at each school 
Group work and discussion 




10* 


Presentation of 
training plan 


Issues on management 
of training in a school 


Closing 

ceremony 










Discussion 


Discussion 











Table 1 . The training program for a 10 days training course 



4 Results of differences among pre, post and after taking the course 

The rating system consisted of four levels: 4 (good understanding), 3 (some understanding), 2 (little 
understanding), and 1 (no understanding at all). The maximum score is 424 (calculated by 4 times 106), and 
the minimum score is 106. Figure 4 shows the change of total scores with post- and after check sorted on 
pre-check scores. This figure shows that the scores with post- and after check are almost independent on 
pre-check scores, which means the training program was effective regardless of participants’ prior 
knowledge and skills. 

The figure also shows three regions which are the pre-achieved region before training, the achieved region 
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by the training, and the unachieved region in spite of the training. Therefore, we can evaluate the area size 
of the achieved region by the training as the training effectiveness. In roughly estimation, it was as desirable 
as expected. 

4.1. Analysis of factors affecting the training effectiveness 

In order to pick up the factors affecting the training effectiveness, we made the following assumptions. 

Assumption 1. The in-service training system will be effective to all participants with various prior ICT 
knowledge and skills, as anticipated from the previous section. . 

Assumption 2. The Web-based training support system and CD-ROM materials will work well with 
comparison to the traditional training system such as lecture-based instructional style. 

The model for testing these assumptions is as following. 

Post-check score <- pre-check score + Web-based support system 
+ CD-ROM materials + instructional style 

Where, pre-check score represents participants’ prior knowledge and skills, post-check score represents 
participants’ knowledge and skills achieved by the training, and instructional style shows instructor’s 
presentation, participants’ individual learning, group works and so on. This model is based on the relation 
that the post-check score, it is called as a dependent variable, is influenced by other independent variables 
such as pre-check score, Web-based support system, CD-ROM materials and instructional style. Here, we 
call the above independent variables as factors affecting the training effectiveness 



424 



318 



post - 
check 
score 

212 



106 



Figure 4. Dependence of post- and after check scores sorted by pre-check scores 

The regression analysis should be suitable for evaluating the factor’s size. The factor’s size can be obtained 
by calculating the regression coefficients of each independent variable in the above regression equation. 
Here, we will adopt the time spent during the training course as a value of independent variables except pre- 
check score. This means that the factor with more spent time contributes to the training effectiveness more 
than the less spent time. We can estimate the size of regression coefficients as the affecting size of factors. 
Time spent in each factor was measured using videotapes recording all training course. A part of data is 
shown in table 2. In the table, topics number shows topics in the training course, 12 in total, factors show 
lecture, presentation, group work Web-based support, CD-ROM for use and so on, and numerical data in the 
cell show the spent time in each factor and each topics with the unit minutes except scores. 

Table 3 summarized the result, data of the regression coefficients’ size, calculated by regression analysis and 




the result is shown in figure 5 as a visual graph 
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Table 2. A part of data table showing the spent time for elementary teachers 
Time unit: minutes, pre- and post- scores: average score 



N (total counts of items)=36 (=12 factors times 3 locations) 

□ (decision coefficient) □ 0.713 * (p<0.05) 

Dependent variable □ scores of post- check 

Factors (Independent variables) □ 

factors sub factors regression coefficient 

(1) pre- ski 11s of teachers pre-check score 0.010 

(2) training style lecture .0.332 + (p<0.10) 







presentation 


□ 0.220 






group work 


0.180 






individual work 


0.364 + (p<0.10) 


( 3 ) 


supporting system 


web-based system 


0.589 ** (p<0.01) 


( 4 ) 


CD-ROM materials 


for self learning 


0.310 






for group work 


0.068 






for presentation 


0.010 


( 5 ) 


school class 


elementary 


0.360 + (p<0.10) 






junior high school 


0.065 



Table 3. Result of regression analysis: coefficient’s size of each variable 

From table 3 and figure 5, we could find the following results. 

(1) The biggest influencing factor was the Web-based support system, which was statistically significant 
within 1 percent level. 

(2) More spent time using CD-ROM for self-learning contributed to higher scores in the post-check. This 
means that the CD-ROM is more effective when used for self-study than other uses such as group work 
and instructor’s presentation. 

(3) More spent time in instructor’s presentation was less effective to the scores of post-check, which was 
statistically significant. This means that a traditional teaching style, such as one way directed lecture- 
based style, is ineffective to the training. 

(4) Differences were found between teachers of elementary and junior high schools. 

(5) Scores of post-check were almost independent to scores of pre-check. This means that the training 
system will work effectively regardless of quantity or quality of a teacher’s prior knowledge and skills as 
expected in the previous section. 

From the above results, we can conclude that the first and second assumptions substantiated. 
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4.2. Analysis of factors affecting on the works produced by participants 

In the training course, participants were assigned to produce works such as homepage, a lesson plan using 
CD-ROM, and so on. Instructors rate the produced works with four grades from the points of check items’ 
view. The regression analysis was done using the similar method as the previous section. The result is shown 
in table 4, 



web -based support 
individual learn mg 



group work 





zir 




presentation 



CD- ROM iel£ teaming . 



elementary 



CD-ROM presentation 

JSM 

CD-ROM group wort 









junior high 





pre-check 

score 



presentotion 



group work 



individual 

leorning 



ireb-bosed 

support 



CD-flOM 
sell leorning 



CD-flOM 
group work 



CD-KOM 

presentotion 



elemenlory 



junior high 



- 0.010 



- 0.332 



- 0.220 



0.180 



0.364 



0.589 



0.310 



- 0.068 



0.010 



0.360 



0.065 



Figure 5. Comparison of each factor’s size 



N (total counts of items)=21 (=7 produced works times 3 locations) 
□ (decision coefficient^ 0.764 ** ( p<0.01 ) 

Dependent variable □ scores of produced works rated by instructors 
Independent variable □ 

factors sub factors regression coefficient 

(4) CD-ROM materials for self learning 0.522 ** (p<0.01) 

for presentation □ 0.574 ** (p<0.01) 



Table 4. Result of regression analysis on produced works 

We could find only the factor of CD-ROM materials as statistically significant, shown in table 4. From this 
result, we could find that only using CD-ROM materials for self-learning works effectively. In contract, the 
use of CD-ROM materials for instructor’s presentation was relatively ineffective. It can be interpreted that 
one way directed instructional style was also ineffective to work-production, which was the same finding as 
shown in the previous section. 



5 Conclusions 

The conclusion is summarized as the following: 

(1) Web-based training support system and CR-ROM materials were effective to improvement of teacher’s 
knowledge and skills, regardless of quantities and qualities of prior knowledge and skills. 

(2) Traditional training style, one-way directed and instructor-centered style, was ineffective to in-service 
training. 

(3) Using CD-ROM materials for self-learning enhanced the quality of teachers productions. 

From the above conclusions, we can give some useful suggestions to educators and administrators who plan 
to conduct in-service training courses. The important points are: 
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(1) Modify the training approach from a traditional teacher-centered to a participant- centered style (refs. [2]), 

(2) To introduce rich materials for self-learning or group works (refs [4]), 

(3) To introduce Web-based support system, 

(4) To promote opportunities for exchanging teachers’ knowledge and skills, especially rich-experienced 
practical know-how. 

Suggestions (3) and (4) are very important because teachers can share their know-how with others, which 
works well when conducting real lessons in schools (refs. [9]). Web-based support system enables teachers 
to share their know-how and knowledge anytime, anywhere, and with anyone. Moreover, CD-ROM 
materials also can be used at anytime, anywhere, and by anyone. Therefore, it is important to provide rich 
self-learning resources that contain expert teachers’ case studies and a support system that enables the 
sharing of dynamic, experience-based knowledge, and communication. Such resources can truly be called 
integrated training environments. 
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Management System 
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This paper describes a LAN-based module program for formative evaluation 
during classroom teaching. Within one-hour classroom instruction, teachers can 
hardly assess what students significantly understand during history instruction. 

With this system, teachers are apt to create, modify, retrieve and manage 
formative evaluation processes based on a database. In addition, the teachers 
organize test items and other instructional materials in elaborated sequences 
relevant to their teaching plans. Our system consists of an instructional 
management tool and a module -based formative evaluation system. Our system 
is implemented with Visual Basic 6.0, Access DB and Crystal Reports 5.0. The 
final part of this paper presents how to enhance system based on the Bayesian 
inference system. 

Keywords: formative evaluation, knowledge diagnosis, instructional 

management. 



1 Introduction 

School teachers are either implicitly or explicitly affected by the use of computer in classrooms. They tend to 
consider utilizing the computing power for their instructional improvement. One of the most prominent 
purposes for teachers to adopt the computer is regarded as Computer-Managed Instruction (CMI). How to 
manage instruction and learning in real classrooms turns out to be a very practical problem for teachers [2]. 

The survey result conducted by Lee, et al. [4] indicates that over 60% of the participants responded that they 
prefer using one computer per each person when software is applied to classroom instruction. Even though 
web technology has been dominated under the name of WBI (Web-based Instruction), our system is 
designed on the LAN environment due to heavy network traffic and rather a weak requirement for multimedia. 
We only focus how to enhance the teachers’ capability to manage their instruction by adding formative 
evaluation module into the existing instructional sequences similar to a web-based formative assessment 
program [1]. 

This paper describes LAN-based VisEval which is a module -based formative evaluation system. The VisEval 
was originally designed as a quiz-like testing tool that makes Korean teachers avoid difficulties in assessing 
teaching materials to students. With this system, teachers are apt to evaluate the students’ understandings 
as a part of performance evaluation that is encouraged among most of the Korean teachers by the Ministry of 
Education. In addition, the VisEval helps teachers elaborate teaching sequences relevant to their teaching 
plans by simply adopting other teaching resources due to modularized design of VisEval. 



2 Design and implementation of VisEval 1.0 
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The VisEval 1.0 is a prototype system that helps teachers easily check students' learning progresses. This 
prototype is aimed at playing the role of an evaluation aid for teachers who usually skip the evaluation 
procedure during classroom teaching. The teachers thus take advantages of using the VisEval by creating, 
classifying, retrieving and managing test items before instruction. The overall framework was implemented in 
Visual Basic 6.0 with Access DB for Windows environment. 

Within the limited time for instruction, teachers are expected to motivate students’ learning and control the 
overall pace of instruction. Without any tool, teachers can hardly try to evaluate quiz-like tests at the end of 
instruction with further statistical analysis as a part of performance evaluation. With VisEval, teachers can set 
up the test duration so that the main classroom teaching cannot be overloaded for both teacher and students. 

Our system is based on Module program which means that the whole teaching sequence are divided into 
small parts so that the teacher can organize the necessary parts for lesson plans. The traditional CA1 program, 
on the other hand, is regarded as system-driven program, which means that the whole instructional program 
consists of one system containing domain knowledge and control structures. Our system is thus treated as a 
component-like module-driven program. 



2.1 Module for teachers 

A module for teachers is designed in the following ways: it is easy for teachers to reorganize test items and to 
manage evaluation procedures. The other important factor is associated with the way feedback is provided. 
Our system only provides immediate feedback only when the active engagement of the user is assured. Once 
the feedback is issued, the user is not allowed to solve similar problems but to receive guided instruction 
from the system with the treatment of the whole test items again. The user is only allowed to solve multiple 
choice items at a time. According to the result, the user is expected to repeat the test items in a linear fashion 
since the teacher only provides 10 minutes or so for students to transfer acquired knowledge items. The table 
1 shows the main functions of instructional module of VisEval. 



Table 1. Functions implemented in the module for teachers 



Function name 


Content 


Item manger 


Generation, modification, deletion of item DB, 
conversion of DB items into texts 


Responses analyzer 


Item response analyzer, Scoring functions 


Simulation for students 


Student’ s trial for simulating 


Other tools 


Distribution of test files on LAN, collection of the test 
results, sharing of test files in different formats 



2.2 Module for students 

Once the module for teachers is ready to go students are expected to login the system. The immediate 
transaction of the test items enables the students to assess their own understandings of the learning 
materials. Without the corrective feedback, each student is allowed to follow his/her own guided instructional 
module provided by the system. Different formats of the test items are available with special characters such 
as mathematical symbols implemented by RTF (Rich Text Format). After the evaluation procedure, the 
transaction of the test results to the teacher’ s computer is finished via LAN. The sample result of the 
students’ assessment for one class is shown in figure 1, being created from Crystal Reports for Visual Basic. 
The analysis of each test item for each class is processed from the module for teachers as depicted in figure 2. 

As the module for students is designed to provide supplementary learning according to test results, the 
control flow for guided learning is implemented. This mechanism is up to present indexed by the teacher’ s 
setting with a predetermined threshold. We are currently investigating much fine-grained knowledge 
diagnosis method such as knowledge-state inference system based on Bayesian inference network. The flow 
control for guided learning to date b coded using item-binary matrix relying on the teachers’ previous 
teaching experiences. The future modification plan of our system is explained in the following section. 
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Figure 1. Test results taken by individual student in each class 




Figure 2. Item responses for students in each class 



3 Conclusion 

The features of the VisEval are, up to present, suited for Korean teachers’ needs of easily evaluating high 
school history instruction in the computer rooms. Even though there are some limitations as being an 
automated instructional system, the VisEval will be enhanced further as the system evolves. One of the 
drawbacks in our system is the lack of guides and suggestions on how to accurately diagnose the students’ 
knowledge states. The following future plan dictates possible enhancement of our system with two extra 
modules based on the previous research [3]. 

The first part is a module that draws knowledge inferences for a given student’s problem states based on 
pre -built test item attributes. The second part decides which abilities of the student need complementary 
practices according to the diagnosis done by the inference system. This second module also provides 
students with adaptive ways of taking relevant problem items mapped from item-attribute matrix in the domain 
of middle school mathematics. As shown in Jun’s research [2], all of these mechanisms are operated on the 
WWW environment via Common Gateway Interface (CGI). 

We begin with carefully examining test items in terms of content domains and student abilities. By 
constructing an incidence matrix with content domains and student abilities, each test item can be mapped 
into a content-ability matrix. With this test item matrix, we proceed to building knowledge -state inference 
system based on Bayesian Inference Network algorithm [5], 

For example. Certain contents and abilities can be pre-requisites of other concept, say node Nl, which means 
students first need to understand the pre-requisites of the concept (Nl), Priors of network nodes are 
empirically assigned according to teachers’ teaching experiences. This big graph is then fed for computer 
programs that calculate Bayesian inference [4], We tested Ergo [7] and Netica 
(http://www.norsys.com/home.html) for WWW environment. The result of processing such a Bayesian 
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program updates probability distribution among nodes. Such posterior probabilities are discretely scaled so 
that VisEval can provide fine-grained guided learning materials to students. 

In short, VisEval has several possibilities for future enhancement on WWW environment as an instructional 
management system for distance education as similarly reported in [1]. We furthermore pursue a fine-grained 
knowledge diagnosis capability with which the system intelligently guides students on their own paces 
outside classrooms. The use of Bayesian network algorithm seems promising for our future project. Error 
diagnosis system such as Rule Space model is another reference to encode test item attributes and projection 
space of multi-facets of knowledge items [9]. Even though the scope of our system is mainly focused on 
teachers’ side in classroom teaching, the future version of VisEval can extend its capability of guiding 
students as an individualized learning management system. 
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This paper describes the development and operation of a Web-based system to 
support computer exercises used in a course on data structures and algorithms. 
To develop such a system, this paper proposes using the functions of a Web- 
based system to deal with a learner* s state transition model based on computer 
exercises. The Web system developed by us has useful functions, some of 
which are the management of participant registration, identification of learner’s 
goals, web service of exercises, mutual interaction between participant and 
teacher, management of report submissions, and both provision and analysis of 
electronic questionnaires to participants. The use of this system resulted in 
students’ heightened motivation to work, good communication between 
participants and teachers, and a reduced workload for teachers. 

Keywords: Web, Database, Exercise, Autonomous Learning, Domain Model, 
Communication, Questionnaire, Data Mining, Operation, Evaluation 



1 Introduction 

The new curriculum of the Department of Intelligent Systems at Hiroshima City University has added 
computer exercises to subjects related to algorithms and programming, thus encouraging students, from 
freshmen to sophomores, to make the most of their ability for practical programming with representative 
algorithms. The curriculum offers two ongoing three-hour courses that include theory and practice. 

This paper focuses on computer exercises for the course ’’Data Structures and Algorithms which is a part of 
the core curriculum for sophomore students. The general objective of the course [1,2] is to facilitate the 
transition from computer literacy to a professional level of information processing. Even though students 
have considerable knowledge of computer operations, they do not have perfect command of them 
Moreover, they do not have enough experience in basic programming techniques. In order for them to have 
command of the theory and the practice, we have developed many exercises to improve the management of 
participant registration and learner's goals, information about the exercises, mutual interaction between 
participants and teachers, management of report submissions, and collection of questionnaires, among others. 
However, a problem arises because the workload for both teacher and students increases in the process. To 
solve this problem, we have developed the necessary support Web system dealing with a learner's state 
transition model based on computer exercises. Moreover, we report the operational results obtained from 
real exercises . 



2 Assessment of learners’ situation before the training 

The contents of the courses "Data Structures and Algorithms I" and "Data Structures and Algorithms II" 
were divided into two courses, each including both theory and practice, using C in the new curriculum The 
former includes major elements such as stack, queue, list, naive sort, recursive function, quick sort, tree 
structure, and binary sort in the second semester of the first year. The latter includes major elements [3, 4] 
such as complexity, file processing, linear search, binary search, hash, B-tree, pattern matching, graphical 



69 






searches, Kruskal, and Dijkstra in the first semester of the second year. Since students can easily understand 
the content of many classes if they have attended C in an earlier semester, "Structured Programming" was 
also organized into two courses including both theory and practice using C in the first semester of the first 
year. This course includes major expressions such as if-, while-, and for-statements, array, data types, 
pointer, function, and structure in C. Moreover, the teaching of computer literacy includes major elements 
such as word processors (e.g., LaTex), programming tools (e.g., mule, e-macs), drawing tools (e.g., TGIF), 
the input tool for Japanese characters, electronic mail, X-window, and the shell command on UNIX, among 
others, in the same semester. 

An evaluation of the learners’ situation before starting 
the course "Data Structures and Algorithms II" that is 
the focus of this paper provided the following results: 

(1) Students did not have much knowledge about 
algorithms and data structures with practical usage. 

They had learned simple and short programs but 
did not have much experience with longer 
programs. For example, they did not have 
experience in how to update longer programs by 
themselves. 

(2) They did not have enough motivation for 
autonomous learning. They were less eager to 
learn than freshmen. For example, they did not 
consult textbooks or dictionaries on their own 
when they had trouble understanding an exercise. 

(3) Twenty-five percent of the students did not understand the C language. Seventy-five percent of the 
students tended to forget the C language, since they had not had a chance to practice it for more than 2 
months after the second semester of the first year. 

(4) Many students did not have sufficient skills to attain perfect command of software tools such as TGIF 
or LaTex. 



3 Conceptual view of the computer exercise 

Figure 1 shows the system configuration to support the exercise. Since each learner does his exercises at a 
workstation connected to the Internet, he can access information managed by the Web server. The Web 
server stores the exercises as HTML documents. The application program located in the CGI (Common 
Gateway Interface) manages information related to his registration, personal goals, and questionnaires. The 
application program is implemented in Pari, Shell, and SQL. The information inputted by the Web browsers 
is stored in the database and used by the learners. 





Figure 1. System Configuration 




70 



We tried to computerize human work as much as possible in the existing computer exercise. Notice of all 15 
exercises included in the course was given on the Web page. We connected both basic programs and 
measurement data to the Web page for each exercise. Using a Web browser, both could be downloaded from 
the Web server to a student’ s site. Before starting on the first exercise, students had to fill out an electronic 
registration form for the class using the Web browser. When a student inputted his school number, name, 
password, and e-mail (electronic mail) address in the registration form, the system issued him a registration 
number using e-mail and the Web page. If the student needed any information about the exercises after that, 
he could get it by inputting his registration number and password using the Web browser. 



Figure 2 represents the state 
transition for the computer exercise 
model. "Starting the Course," 
located at the left side of Figure 2, 
represents the state before starting 
the class. The student moves to the 
state of "Completing the Course" if 
he finishes all exercises successfully. 
If the student inputs personal data in 
the class registration form, the 
student moves to the state of "Class 
Participant." If the participant replies 
to the first questionnaire and inputs 
his personal goals for the exercise 
using the Web browser, he moves on 
to the state of "Exercising." At this 
stage, the learner is allowed to solve 
the exercise. If the learner inputs a 
question to the teacher on the Web 
page, he receives a reply from the 
teacher on the Web page. After 
finishing the exercise, the learner 
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Figure 3. An Example of System Operation 



moves on to the state of "Making the Report" and can answer our questionnaire for the exercise as he 
finishes the exercise. If the learner submits his report to the teacher, he moves on to the state of "Waiting for 
the Evaluation." If the evaluation is poor, the teacher contacts the student, helps him, and asks him to re- 
submit the exercise. The Web system does not support their interaction in the situation, since we believe 
that face-to-face communication is preferable. This situation is different form Fujimoto's Classroom 
Management System [5]. After the learner reaches the state of "Completing the Submission," he will input 
his personal goals for the next exercise. After that, he will move to the state of" Exercising." 



We place great importance on the use of educational methods [6, 7] including "Reading, Writing, and Using 
an Abacus" to achieve the goal of "autonomous learning and thinking." For students belonging to the 
categories (l)-{4) mentioned above, the computer exercise model includes the following educational 
methods. Students in ( 1 ) and (3) are asked to read longer programs downloaded from the Web server, write 
the respective flowchart, update the subparts, and measure their performance in the state of "Exercising" 
shown in Figure 2. 



Students in (2) are asked to define their personal goals before reaching the state of "Exercising" and write a 
self-evaluation in the state of "Making the Report." In the state of "Exercising," students are given an 
ambiguous exercise to learn the value of searching for information. In this way, students are encouraged to 
develop their creativity skills. Moreover, students are strongly advised to use textbooks and dictionaries if 
they have unresolved questions. Students in (4) are strongly encouraged to use such tools as TGIF and 
LaTex when preparing a report that includes figures and text. We believe that longer programs particularly 
enhance their proficiency in using tools. In order to determine an accurate grade for each exercise, we 
evaluate the reports submitted by the students and their answers to the questionnaires. Since we receive the 
results of the questionnaires immediately through the Web, we use such results to improve the exercises and 
coach the students. Moreover, the students can also receive their scores in a very short time . Students can 
compare each other’ s scores if they are given access to the statistics. Giving students access to the statistics 
is regarded as the key to ensuring an environment of awareness [8]. 
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4 The results of system operation 



Figure 3 represents an example of the operation of the system. Web page number (1) in the figure relates to 
the state of "Starting the Course." Page (2) is the class registration form. Page (3) gives anchors for 
information about all 15 exercises included in the course. If a learner selects one of the exercises on the 
page, he can use the exercise page (4). He can access his record of submissions and re -submissions using 
Web page (5). After inputting his personal goals using Web page (6), he moves on to the state of 
"Exercising." When he finishes the exercise, he moves on to the state of "Making the Report" and inputs the 
questionnaire on Web page (7). The results of the questionnaires are immediately stored in the database. 
Not only the teacher but also the learners are able to compute the statistics of the results from the database in 
real time. Page (8) relates to the statistics. Pages (9) and (10) are for teachers' use only. In page (9), each 
student has 15 check boxes, each divided into an upper and a lower section. If the report evaluation is good 
in the state of "Waiting for the Evaluation," the teacher puts a checkmark in the upper check box. If not, he 
puts the checkmark in the lower check box and helps the student so that he re-submits his work. Page (10) is 
useful for analyzing questionnaires stored in the database. The analysis includes the method of data mining 
[9] implemented in SQL. 

Application of the system operation started at the Department of Hiroshima City University in April 1999. 
This system motivates students to do their exercises, provides good communication between participants and 
teachers, and reduces teachers' workload. The evaluation results of questionnaires and examinations related 
to the exercises are as follows: 

( 1 ) Ninety percent of students studied for 0.5-2.0 hours at their homes and were interested in the lecture. 

(2) Twenty-six percent of students spent less than 2.0 hours preparing the report and exercising, 53% spent 
2.0-5. 0 hours, and 21% spent more than 5.0 hours, not including class work. 

(3) Seventy percent of the 12 students (25%) previously mentioned understood the C language. Moreover, 
all students made progress in their studies. 

(4) Ninety-five percent of the students reported good understanding of the algorithms used in the exercises. 
Eighty-seven percent of the students passed the examinations. 

(5) The students acquired good skills at using TGIF, LaTex, and other programs to write reports. 

(6) Seventy percent of the students felt that the teacher did his best in the classroom, and 17% of them 
barely approved of his performance. 

5 Conclusions 

We proposed a computer exercise model for the course of "Data Sructures and Algorithms II" and 
developed a Web support system for computer exercises using the model. We place great importance on 
educational methods including "Reading, Writing, and Using an Abacus" so that our students acquire the 
skills of "autonomous learning and thinking." Computer exercises using the Web system give students a 
chance to enhance their capabilities of "autonomous learning and thinking" and "creativity." The system run 
on the Web server has useful functions, some of which are the nanagement of participant registration, 
identification of learner's goals, web service of exercises, mutual interaction between participants and 
teachers, management of report submissions, and both provision and analysis of electronic questionnaires to 
participants. The use of the system resulted in students’ motivation to do the exercises, good 
communication between participants and teachers, and a reduction of teachers' workload. In order to 
achieve more concrete results, the students studied more at home and were enthusiastic about doing their 
exercises. Moreover, the students learned how to make a report using TGIF, LaTe* and other programs . 
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Research on the effects of hypermedia learning environments often suffers from 
a lack of systematic control of learning conditions, especially the sequencing of 
the content. While available tools for logfile analysis are confined to delivering 
frequencies and other figures, the tool to be presented (EDASEQ: Exploratory 
Data Analysis for Sequential Data) was developed to facilitate the analysis of 
the navigation paths of single learners as well as “average” paths of a group of 
learners. Because standard statistical procedures for handling sequential data are 
not suitable here, the tool is primarily founded on graphical methods. Navigation 
processes are represented by transition matrices, and with additional 
visualizations and trajectories. Apart from descriptive portrayals, the tool also 
allows for categorizing empirically found navigation patterns on the basis of 
theoretically defined prototypical patterns. Furthermore, it is possible to 
compare the patterns of single learners or groups. Results can be used to better 
explain the effects of self-regulated learning in hypermedia learning 
environments. Without knowing variables like sequencing, time -on-task, or the 
number and configuration of examples studied by learners, it is hardly possible 
to interpret the impact of external learning conditions on the learning outcomes. 

Keywords learning processes, navigation, hypermedia, methodology, tool 



1 Introduction 

A fundamental problem of research on self-regulated learning is the possible variation of learners’ behaviors, 
especially regarding variables like the sequence of content, the time spent with studying different parts of 
subject matter, the number and arrangement of examples and exercises: Have they really worked through all 
the relevant information? How many examples were chosen, with what methods of representation and in 
what combinations? How many exercises were worked through, and to what result, resp. with mistakes of 
what kind? How long were the learners occupied with what contents? This is valid for every type of research 
on self-regulated learning, but especially for learning with hypermedia. Even with the same context 
conditions, quantitatively and qualitatively completely different courses of learning are possible and thus, in 
consequence, very different results. Even when the learner activities displayed are described exactly, there 
are differences with respect to the quality of the elaborative treatment; the external conditions of the learning 
processes, however, are principally controllable. Unfortunately, standard statistical procedures are not 
suitable to represent an “average path” in an educational hypermedia system: Mean times spent on looking 
to specific pages or mean frequencies of visits are often not sufficient to explain differences in learning 
outcomes. 



2 Aims of the development of EDASEQ 1 

For the description and categorization of such processes there thus remain graph theoretical procedures. 
There were already attempts at implementing these some time ago; the best known is probably Flanders’ 
(1970) procedure for the analysis of teaching (cf. also Canter, Rivers & Storrs, 1985). For the treatment and 



1 Supported by the Deutsche Forschungsgemeinschaft (German Research Foundation), grant Az. Ho 
649/9-1,2,3). 
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evaluation of data on the basis of graph theoretical considerations there is, however, as yet no method which 
is relatively simple to use. It was therefore an aim of the development of methods within the framework of a 
six year research project on case-based hypermedia learning environments in vocational training to develop 
a procedure allowing recorded data of learning processes to be so prepared that a) a data reduction is brought 
about which allows those processes to be analyzed exploratively, b) a categorization of typical processes 
becomes possible, c) records of several processes can be aggregated and finally d). Comparisons are made 
possible between (1) single records of processes, in order to see differences and similarities, (2) an 
individual process and an aggregated one, in order to be able to study deviations of single learners from the 
typical route taken in their group, (3) two aggregated processes, in order to compare groups under different 
treatment resp. marginal conditions using experimental designs, (4) a single or an aggregate process with a 
prototype process that is produced artificially, in order to categorize processes of one or several prototypes 
within the framework of defined deviations, or to test hypotheses. Apart from the characteristics of the 
processes, it should also be possible to extract simple statistics: e.g. frequency of the calling up of specific 
screen pages, specific transition frequencies, length of stay etc. 



3 Forms of representation 

In order to represent hypermedia navigation processes, there are first of all two different but mathematically 
equivalent codes: transition matrices on the one hand and aligned graphs on the other. Whilst one can see 
conspicuous characteristics in the graphical representation, the matrix representation allows the calculation 
of indices. Since both forms of representation are practicable, both should be taken into consideration. One 
special feature of well designed hypermedia learning systems is a structured presentation of knowledge 
given in such a way that learners have the choice of either informing themselves superficially or of going 
deeper into the subject at any chosen place, or of conbining both courses of action: first gaining an overview, 
then deepening their knowledge. In order to determine the extent of the "deepening” - assuming an 
appropriate structuring of knowledge in the medium -, two characteristic values, the mean “depth of 
elaboration” and the “variance of elaboration” have been developed. The depth of elaboration is a rating for 
every hypermedia occurrence, which is all the higher, the deeper the corresponding screen page goes into a 
specific subject. If, for instance, the highest level with the index number 1 is the term "statistics", then pages 
on the subject "inference statistics" or "descriptive statistics" would have the index number 2 and a page on 
the subject "log-linear models" would have, for example, the index number 5. The arithmetic mean of the 
values of all screen pages visited could then give an indication of the extent of the "deepening” or 
"elaboration" of the material; the measurement is completed by the elaboration variance ascertained 
analogously. Not least, characteristics of the chronological process should be portrayable. 



4 Realization 

As the first step towards a reduction of process data in the Mannheim research project "Case-based learning 
problem" - in compliance with the demands - a software -technical evaluation procedure was developed. This 
enables processes to be transferred rapidly into transition matrices, so that firstly the simple frequencies of 
the consultation of specific pieces of information and of the transitions between offers of information can be 
ascertained. The learning programs developed in this project each encompass approx. 150 screen pages; 
learners need up to five hours to complete the given tasks and corresponding records comprise 3000 - 5000 
single entries, each consisting of the time (in seconds after midnight) and the designation of the respective 
screen page. Log-files existing as ASCII text files are downloaded and converted into MS Excel files. For 
the simultaneous treatment of a larger amount of records it is also possible to stack them. Process data in the 
form of transition matrices can furthermore be compared to each other and also aggregated. In order to 
reduce data one can also stipulate that transitions which are more seldom than a specified threshold value 
should be ignored. 



5 Examples of process representations and indices 

The following representations are based on fictitious data; i.e. records were produced with the specific aim 
of representing certain processes, in order to determine whether the corresponding characteristics are 
perceptible. Apart from this the size of these records was to be restricted, in order to enable a written account 
to be given, shows the evaluation as regards the frequency with which single screen pages were called up, as 
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well as the length of the stay there (absolute in seconds and relative to the complete time needed). Ills. 1 
shows a transition matrix with aligned graphs of the process included (option). One alternative graphical 
representation (’’chronological") is given in 111. 2. Here, above all, recourses to previous steps are clearly to 
be seen: the test person would, in this case, have chosen a strategy whereby he/she began by choosing page 
B1 on the higher level, "deepening” from there straight to BX1, going back to Bl, choosing mother 
"deepening" (BY2) etc. The values in the main diagonal indicate how many time units the learner has here 
stayed on each separate page. 

In a third, more concise representation of the process every node (page, screen, chapter etc.) is represented 
by one cell and the navigation process is shown by arrows between the cells. Analyzing aggregated data, the 
thickness of the arrows indicates the frequency of the transitions between two nodes. So, a sequence of thick 
arrows represent a “modal path”, i.e. a path used by many users. (111. 3) 




111. 1 : Transition matrix with process graph included (Al, Bl etc. indicate screen pages) 




111. 2: Linear representation of the process (Al, Bl etc. indicate screen pages) 
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6 Conclusion 



The procedure which has been developed is first of all explorative, i.e. data are so prepared and represented 
that they allow categorizations and comparisons, thus offering a basis for the forming of hypotheses. Very 
extensive record files, in particular, are reduced. Although the procedure for the analysis of records on 
navigation was developed in hypertext, resp. hypermedia systems, it is also suitable for the treatment and 
analysis of data on the observation of teaching or other courses of communication. 
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Following constructionist principles, postgraduate students who were studying a 
paper on Human Computer Interaction were required to build educational multimedia 
systems and then to evaluate those produced by their colleagues. The experience of 
developing a multimedia system, together with lectures and access to general material 
on the topic, enabled them to provide valuable insights into important issues. 

Nonetheless, the students were not, on the whole, able to transfer all that they had 
learned when building their own systems into an evaluation framework. The 
provision of scaffolding was recommended to facilitate transfer. 

Keywords: Multimedia, Evaluation, Constructionism 

1 Introduction 

What are the criteria that should be used to judge the effectiveness of interfaces for multimedia tutorial systems? 
In an experiment with a class of postgraduate students studying Human Computer Interaction (HCI), they were 
asked to develop their own framework for evaluation. To give them some notion o f what to look for and what to 
expect, they first had to form groups and construct their own multimedia tutorial systems. This approach is based 
on the idea of constructionism [6]. By collaborating in a group to develop some appropriate product, it is 
suggested that learners can come to a better understanding of the principles of a subject rather than by just being 
given the information by a teacher. This is in line with the wry comment from Jonassen et al [9] that the people 
who leam most from instructional materials are the designers. 

The students were required to work together with one or two other classmates to produce their own small scale 
multimedia educational systems. Using the knowledge and experience they had gained, they then had to 
individually evaluate the interfaces of the other systems. There was no detailed specification about how to carry 
out these activities. The students had, however, been exposed to the main issues through lectures and 
discussions. They also had appropriate readings made available to them. The intention, therefore, was to see 
what the students, themselves, considered appropriate ways of evaluation in the light of their background and 
their experience in developing multimedia software. An assessment was then made of how much they had 
learned when building systems and how well the knowledge was applied to evaluating the systems of others. 

2 Previous work 

Interface evaluation can be carried out for many different purposes. The distinction is usually made between 
formative evaluation where improvements to a system under development can be determined and summative 
evaluation which assesses the overall performance [8]. There are a range of methods that can be used depending 
on the purpose of the study. Preece [12] categorises these purposes as analytic, expert, observational, survey 
and experimental. Analytic techniques are used to determine the complexity of the interfaces. Expert evaluation 
involves inviting people experienced with interface issues to identify usability problems. Observational, survey 
and experimental studies all have in common the involvement of what Preece terms “Real users.” Users can be 
observed using software, provide feedback about the system through interviews or questionnaires or take part in 
experiments to test the impact of various features of the interface. 

A common method of evaluation that does not involve users is expert evaluation where, as noted above, people 






with some knowledge of interface issues detect possible problems. This process can be conducted in accordance 
with the guidelines formulated by Nielsen and Molich [11]. The following aspects of the interface are all 
considered in what is referred to by these authors as a heuristic evaluation: simple and natural dialogue; speaking 
the user’ s language; minimising user memory load; consistency; feedback; clearly marked exits; short cuts; good 
error messages; prevention of errors, and help and documentation. 

An examination of the literature on multimedia reveals little mention of evaluation. Testing is usually discussed 
but not evaluation [4, 15]. Some important principles emerge, however. Alty [1, p33] points out that “A key 
question is when to use which media and in what combination to achieve the maximum effect.” He also observes 
that success in multimedia depends more on the combination of media rather than on the provision of a rich set 
of media. Frater and Paulissen [5] note that interactive tutorials should allow the user to choose the starting point 
and allow the information to be accessed as often as required. They also offer this piece of advice. “Keep in mind 
that multimedia can make learning much more interesting when animation and sound files are used to explain the 
topic. Also a quiz is more fun when set up as a game” [5, p362]. Preece [12] points out that navigation, too, is an 
important consideration in hypertext/multimedia systems. Users, as she notes need to be able to know where they 
are, how they reached that point, where can they go next and how they get there. This as pect of interface design 
is actually covered by the first heuristic of Nielsen and Molich [1 1] which refers to “Simple and natural dialogue.” 
This takes into account navigating through a system. Interface factors in interactive multimedia systems are also 
considered in Reeves and Harmon [13] and Tannenbaum [14]. 

One recent taxonomy in the literature provided by Heller and Martin [7] aims to help students on multimedia 
courses understand the forms of media as well as enabling them to evaluate the work of others. It has two 
dimensions - the media type and the means of expression (elaboration, representation and abstraction). This 
classification shows, for example, that text might be fully elaborated (large chunks of narrative), can be 
abbreviated (represented in bullet points) or might be abstract in nature such as text in a logo. Students are able 
to check whether a medium has been used in an appropriate fashion. As the authors state, though, the taxonomy 
takes no account of the effect of combining several media. Nonetheless, it is useful in focusing on the evaluation 
of each element. Detailed guidelines about how to use each medium, for example text, can also be found in 
Vaughan [ 1 5] and Collins [2] . 

3 Course structure 

“Topics in Human-Computer Interaction” is a single semester paper for postgraduate students in Computer 
Science and Information Systems at Massey University. Most students have already completed a third year 
undergraduate paper “Human-Computer Interaction” in which the underlying theory is presented. These 
students will also have gained some experience in developing interfaces. The aim of this course is to consider 
issues of current interest such as computer supported co-operative work, innovative interfaces, different ways of 
evaluating the interface, multimedia systems and interfaces on the World Wide Web. Teaching is carried out 
through a mixture of lecturing, student seminars, discussion and demonstrations (of software such as Adobe 
Premiere and Macromedia Director). Students have available to them two books of readings which cover the 
material taught. 

One assignment for this paper involved the students working in groups to develop a small multimedia system 
with an educational focus. The groups could choose any appropriate subject. Each student was then asked, 
individually, to evaluate the interfaces to all the other systems. A set of lectures had been given on the topic of 
multimedia including exposure to several life cycles for developing software of this kind. The topic of evaluation 
had also received considerable coverage in lectures and student seminars. Students were aware that interfaces 
can be evaluated for various purposes and in many different ways (for example by heuristic evaluation, interviews, 
questionnaires etc.) 

Guidance cn the life cycle that should be followed to develop the multimedia systems and the method of 
evaluation required was deliberately kept to a minimum. In the light of the teaching on the course and the material 
available to them, students were expected to make their own informed decisions. In particular, it was hoped that 
the students’ own experiences in developing multi-media software would give them some insight into the criteria 
that should be employed when evaluating the interfaces to the other students’ systems. 
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4 The student systems 

Six groups each developed their own multimedia system. The systems were expected to offer instruction to their 
users and be interactive. A brief description of the systems follows. 



Maori Language Tutoring 

This system was designed to help students learn the Maori language. The study material was based on the 
philosophy that Maori be used wherever possible, with visual and aural stimuli to teach the vocabulary. Words 
were introduced via demonstrations using pre-recorded video clips. The system, however, also contained 
explanation in English for students who did not wish to completely immerse themselves in Maori. As well as 
learning new terms, students could choose to review vocabulary or test their comprehension. Maori music aid 
designs were used in this system where the developers thought appropriate. 

Learning the New Zealand Road Code 

A written test on the New Zealand road code has to be passed before learner drivers can take their practical 
driving test. The aim of the road code system was intended to make the learning process more interesting. It was 
believed that by using animation, audio and video, the learning process would be enhanced. The system included 
tutorial material on aspects of the road code (for example, how to overtake or what to do when approaching a 
roundabout) as well as test material. 

Earthquake Disaster System 

The earthquake disaster system was developed to show people how to behave in the event of a serious 
earthquake. It included clips from a video developed by Civil Defence. Topics that were dealt with included 
planning for and coping with an earthquake. The opening screen showed a photograph of the devastation 
caused by a major earthquake. Music and animation chosen to reflect the theme of devastation accompanied the 
photograph. 

Shape Recognition 

The intention of the shape recognition system was to help children learn how to identify both two and three 
dimensional shapes in a lively and interesting way. Sound, animation and graphics were included in order to 
make the system appealing to children. Another goal of the developers was to make the system easy to use. 
There was a particular emphasis on the use of colour which was seen by the developers as making the system 
attractive to the intended users. The opening screen was designed to capture the attention of children with music 
and morphing shapes. 

Introducing the Internet 

This system, as its name implies, was intended to be introductory in nature. Its target group was school children 
who could find out about concepts such as email, newsgroups, file transfer protocol etc. This system made use of 
graphics and sound but also included lengthy textual explanations. As with the shape recognition system, there 
was an emphasis on the use of colour. Ease of navigation was also a major consideration. 

Undergraduate Studies in Computer Science 

This system allows students to find out about the staff and the papers they teach in a Computer Science 
department. Photographs of staff members were included. When browsing through the system, users were able to 
move from a staff page to obtain information about papers taught by the staff member. Contextual information 
about the location of the building where the Computer Science staff were housed was also provided. The opening 
screen of the system showed a picture of the university grounds. Other pictures could also be viewed. 

5 Educational Issues 

Although the course does not deal with issues of computer-based learning, this was the focus of the assignment 
and gave the students some context for the systems they produced. They were expected to choose an approach 
to teaching which was appropriate for the subject that was being taught and that they felt would be effective in a 
multi-media setting. They all propounded he philosophy of their systems during their presentations. It is 
interesting to compare the different approaches that the students chose for their systems and how this was 
reflected in the presentation styles. 
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The Maori language teaching system immerses the student in the subject and attempts to teach by example. As 
noted earlier, the system can be used without reference to English words or phrases. Maori, like various other 
languages such as Japanese is very much bound up in the culture of the people and so this approach seemed 
entirely appropriate. Maori songs, words and phrases in a commentary with accompanying visuals provided a 
backdrop that was both stimulating and educationally appropriate. 

The road code system contains video clips produced by the students themselves which graphically illustrated 
both correct and incorrect procedures to be followed in various situations when driving. This could be regarded 
as teaching by presentation and illustration. 

Like the road code tutorial, the earthquake dis aster system has an emphasis on illustration using video clips and 
contains other factual information in an appropriate form. 

Unlike the previous systems, the shape recognition tutor includes trial and error examples for the student to 
consider. It takes into account the answers the student gives and does not continue until it judges s/he has fully 
understood all the current concepts. It could be regarded as a mastery system from this point of view. 

The internet system contains a great deal of information in a text -based format, but the presentation was 
enhanced with appropriate animations. Material is set out in an simple to follow form and subjects can easily 
navigate around the system to discover what they need to know. 

The undergraduate studies in Computer Science system also allowed students to learn about the department of 
Computer Science in a discovery mode. In some senses this was the package that was the least like a tutorial 
system, since it just provided information in a non-instructional form 

The underlying objective of the assignment was to determine whether or not students had assimilated a 
fundamental principle of HCI - that issues concerning functionality should not be divorced from interface 
concerns. Given the experience of developing a multimedia system, it was hoped that students would take into 
account the educational aims of the system as well as the multimedia features. It was not the object of the exercise, 
however, to see whether effective learning took place. It was expected that some variation of expert evaluation 
would be followed. What was of interest were the criteria that students incorporated into their checklist. Issues it 
was hoped would be addressed (in the light of the literature on this topic) included the following: 
whether the interface reflected the educational objectives of the system; 
the suitability of the media selected; 
the user appeal of the systems; 
the interface concerns; 

evaluating the execution of the various media. 

6 Results 

Every student (thirteen in total) appraised all the systems developed by their colleagues. All the students 
provided a checklist of the criteria used for the evaluation - some were very detailed and others quite brief - from 
thirty items at one end of the scale to five at the other. The two students with the longest checklists evaluated 
whether the system fulfilled its objectives, the selection of multimedia components and the execution of the 
multimedia as well as detail of the interface such as the provision of feedback, ease of navigation etc. There were 
another three quite comprehensive taxonomies which covered many but not all of the relevant issues. Five 
students used Nielsen’ s [10, 11] guidelines for heuristic evaluation without adding to them to deal with the 
educational or multimedia aspects of the systems. The three students with the short checklists had incorporated 
rather broad categories such as ease of use, knowledge presentation, navigation, multimedia concerns and 
quality of knowledge which gave them reasonable but not complete cover of the relevant issues. 

Expert evaluation can be carried out by anyone with appropriate skills and by more than one evaluator. In one 
case, two people evaluated the systems and combined their findings whilst on another occasion the student drew 
up the framework but did not carry out the heuristic evaluation himself. Some students scored the various items 
and averaged the results. This enabled systems to be ranked. Others did not attempt to provide an overall score 
for each system but left the findings to speak for themselves. 
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1. Did students check to see whether the interface reflected the educational objectives of 
the system? 



In total, eight of the students included questions in their checklist which related to the educational nature of the 
system. Three of these explicitly mentioned the educational objectives of the systems under review before 
providing their assessment. 

“This system is a multimedia tutor system designed to aid students in learning the Maori language. 

The system uses both visual and aural stimuli to teach words and concepts . " 

“The system aims to provide information to undergraduate students. ” 

“It aims at helping children to learn a shape through playing which makes learning easy and fun. " 

The eight students who considered the purpose of the system, that is its educational aspect, did not all ask the 
same questions. A variety of issues were covered as follows: 

How does the system consider educational objectives? 

Is the system suitable for intended users? 

Who is the target audience? 

Is the system aimed at the right audience? 

Does the system have a reasonable informational content? 

Is the quality of knowledge sufficient? 

The evaluations included comments such as the following: 

“Good way to teach a student with audio pronouncing the language and seeing the words on the 
screen. ” 

“ Including some information on the properties of the different shapes and showing everyday 
examples of them would make learning the shapes a richer experience. " 

“It does not really seem to be an educational system, more an informative system. " 

“The current system does not seem to have a glossary page, A page for quick lookups and acronyms 
and jargon would probably be helpful. " 

“It might have been good to have an option of telling users what the different shapes look like. " 

Some of the students, however, not only evaluated the systems in accordance with their checklist but also in the 
light of their experience in appraising the programs. They mentioned, therefore, other important criteria in their 
assessments. One student centred her overall assessment around the suitability of a system for its purpose 
although this was not included in her criteria for evaluation. Two other students, also, mentioned educational 
issues such as whether the systems provided adequate content and comprehensible instructions. 

“I had no idea what I needed to do and how the test was being processed. " 

2. Did the students consider whether the mix of multimedia selected was appropriate for 
the stated purposes of the system ? 

Only two students included in the guidelines for evaluation the need to consider whether suitable media were 
selected and used appropriately. One student asked the question “Is the multimedia actually of use and not 
redundant ?” The other student checked that the mix of multimedia was used appropriately. This student noted 
not only occasions when a particular mix of media was ineffective but also when media was missing. 

“The current system seems to rely too much on textual information. Improvements would be to make 
more use of video, diagrams and to provide more navigation options. These changes would give 
the user a more enriching learning experience. " 

Many other relevant comments were made by the other students about an appropriate usage of multimedia, 
although they did not take the issue into account systematically. 
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“Of all the applications reviewed this has the most appeal due to its excellent usage of graphics 
and sound. The main area it could be faulted on is the large textual explanations given but these 
are offset by the following graphical examples . ” 

“It uses multimedia such as sound and text making the system vivid and active. " 

Only one student fell into the trap of believing that a multimedia system had to incorporate all media. He would 
criticise a system that did not include video, for instance. No regard was paid to whether adding video would 
contribute to meeting the goals of the application. 

3. Since educational systems have to be appealing to their users, did the students take this 
factor into account? 

With regard to the appeal of a system, this issue was only expressly considered by four students. Related 
questions were as follows: 

Is the system interesting and fun? 

Does the user find the system visually appealing? 

Has information been presented in an interesting manner? 

Has the system an attractive presentation? 

Comments made by these students include the following: 

“Its creative design of the main menu ... and its appropriate use of the sound medium, make it 
enjoyable to use the system. " 

“There was no splash screen introduction. Whilst this may seem superfluous, good splash screens 
can be used to arouse a user's interest. ” 

Three other students, however, did mention this issue. One of these was the student wh o did not carry out the 
expert evaluation himself. After watching the evaluation (according to Nielsen’ s guidelines as specified), he 
realised that the system he preferred obtained the lowest rating. He proceeded to base his overall assessment of 
the systems on whether they had an interesting and attractive interface. A second student also focused on the 
interest or lack of it in the programs. Of the Maori tutor, she said “ The welcome interface is impressive. The 
background and the music gives me some feeling of Maori culture.” According to her, another system was a 
little bit boring. 

An issue that relates to the appeal or attractiveness of a system is the appropriate use of colour. Four students 
included at least one item in their checklist concerning colour. Questions were as follows: 

Are too few or too many colours used? 

Is the colour in the system beautiful? 

Does the use of colour help to make the displays clear? 

Is the use of colour bad, normal, good or excellent? 

One system was notable for its use of colour and several comments were made about this 

“The very colourful shapes used are appropriate for the school based children as seen as being the 
intended users . " 

“The colour used in the system is beautiful. ” 

“ Good colour choice, relaxing. " 

This was not the only the system to make effective use of colour, however and one student observed in his 
conclusion that no-one made the mistake of using too many colours. 

4. What typical interface factors were considered? 

All of the students checked for at least one well-known interface concern such as consistency, clearly marked 
exits etc. Seven of them specifically included the guidelines for heuristic evaluation by Nielsen and Molich [11] 
or the updated version by Nielsen [10] in their checklist. 
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It was also expected when considering interface issues that the importance of navigation in interactive 
instructional systems should be recognised. It should not be just one more item in a checklist. Eleven of the 13 
students took account of this issue. 

“No stop, rewind or scroll bars for video. " 

" Gives reasonable freedom to navigate backwards and forwards. " 

“Not very flexible , very linear in its execution. ” 

“It is very easy to get “lost " while navigating through the system. No “back” button provided. " 
“Clicking at various places in the window may move you to unexpected screens. " 

“With the test screens there is no title indicating this. " 

“Have no idea what / am supposed to do in the first screen. " 

Four of the students highlighted the importance of navigation. Three incorporated this into their framework as a 
high level criteria. A fourth not only checked how users moved around the system but whether or not the users 
would know where they were in the system. 

5. Did the students evaluate the multimedia components of the system? 

Four students evaluated the execution of the individual media. Two of these assessed the effectiveness of each 
component: video, sound, graphics, text etc. by rating them on a scale. The third student concentrated on text 
and icons. His section on text was quite detailed, checking the length of the sentence, whether it just focused on 
one issue, and whether there was sufficient white space around it. The fourth student checked that the 
multimedia was not ‘‘over the top”. 

“When the system explained the Maori words, text is well organised. " 

“I liked the use of Maori music with the splash screen. ” 

“Liked the introduction - morphing shapes. ” 

“Widely accepted icons are used to aid page-based navigation. ” 

“The background music is excellent. The button clicking sounds great. " 

“Image excellent. When the system first starts, the animation is creative and attractive. " 

7 Discussion 

Reflecting on the results of the assignment, it became clear that learning about multimedia evaluation took place 
at various points in time. Most of the systems developed by the students were stimulating to watch. As 
developers the students were clearly aware of the need to use appropriate media in suitable combinations [1] and 
of the requirement to navigate easily through the system [12]. Some of what they had learned was reflected in the 
checklists that they developed for evaluating the systems of others. There was a difference, however, between 
the criteria specified by students for evaluation and those actually used when making their overall appraisals. 
These sometimes took additional factors into account that had not been included in the stated checklist. The 
experience of evaluating the systems themselves, allowed further learning to take place. It will be the more 
complete list of factors that are considered in the remainder of the discussion since the experience gained from 
carrying out appraisals is important and should not be discounted. 

Eleven students checked to see whether the interface reflected the educational objectives of the system and two 
of these also considered whether the mix of multimedia was appropriate for the stated purposes. All of the 
students considered at least one relevant interface factor (consistency, clearly marked exits , etc). Six of the 
students also realised the need to find out whether or not a system would appeal to users. Four students 
included assessment of media components in their appraisals, however none of their questions showed a deep 
understanding of media issues. 

It was pleasing from an educational perspective that most of the students when carrying out their evaluations 
took account of the functionality of the system. This cannot be divorced from interface considerations as for 
many users the interface is the system and must deliver the appropriate functionality. 

Interface issues, too, were seen as important by all of the students. Of these, 1 1 checked to see whether a user 




could easily navigate around the program. This is an important issue in interactive multimedia systems and was 
recognised as such by the students. Eight of the students carried out a reasonably comprehensive evaluation of 
traditional interface concerns but for five it was rather rudimentary. This was surprising given the emphasis on 
the heuristic evaluation in the undergraduate and post-graduate courses. 

Overall there were only two students whose evaluation was limited to just those interface issues covered by 
Nielsen [10, 11]. This meant that they excluded educational considerations, the appeal of the interface, an 
evaluation of the individual media and whether or not they were used in appropriate combinations. 

A major weakness in the student evaluations’ overall was the failure to consider whether the mix of multimedia 
selected was appropriate for the purpose of the system. Whilst the students did consider educational issues at a 
high level, they found it difficult to move to a detailed perspective, that is were suitable media selected and 
combined? This may involve greater knowledge of the potentialities and problems of the individual media than 
the students possessed. They tended, therefore, to have an overall impression of a system. This was reflected 
again in the failure of two thirds of the students to evaluate the exe cution of each media component. 

Around 50% of the students did not take appeal/interest and fun sufficiently into account. This can possibly be 
attributed to the fact that they were not the intended users of the systems. If they had been drawing up a list of 
questions for users to answer they may have incorporated this. Nonetheless, it was an important omission as 
multimedia systems set out to interest and hold the attention of their users. 

As the above discussion shows, students were particularly weak in considering what was to them the new area of 
multimedia. They did not appear to have the knowledge or experience to determine how to evaluate the media. 
They were given some exposure to these issues in lectures but do not appear to have followed them up. Whilst 
no one student came up with a complete checklist for evaluating multimedia systems, amalgamating the items in 
their checklists enables a comprehensive framework to be developed. See Appendix 1 for the main features of 
this. In future it may be preferable to provide students who have built a multimedia system with some scaffolding 
to help with the evaluation phase. Scaffolding [3] refers to supports that can be provided by a teacher to 
students. The main headings in the taxonomy outlined in the Appendix could be provided. The students could 
then be asked to develop appropriate questions for each area. 

8 Conclusions 

The students learned a great deal by building multimedia software and evaluating the systems of others. This 
was reflected in the perceptive comments of the students made in their written assignments. It was not always 
reflected, however, in the frameworks for evaluation that they developed, only two of which were comprehensive. 
Certain areas were handled well by the students, for example checking that each system was suitable for its 
purpose and the importance of navigation. Two significant issues, though, were only identified by a minority of 
the students - the need to choose appropriate media and to determine how well they had been produced. It 
appears that because the area of multimedia was new to the students, they needed more scaffolding in place to be 
able to learn from their own experiences. Instead of developing an evaluation framework from scratch, some initial 
information can be given to students in future that they then have to flesh out. 
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Appendix 1 

1 . Does the system meet its objectives? 

Who is the target audience? 

Is the system suitable for the target audience? 

Does the system include (in the case of educational systems) sufficient content? 

2. Has an appropriate mix of multimedia been selected? 

Have sound and text been used effectively together? 

Have sound and graphics been used together effectively? 

3. Will the program appeal to users? 

Is the system fun? 

Will the user find the system visually appealing? 

Has the system a features that will pall over time e.g. an unusual sound or joke? 

Has colour been used in an appropriate fashion? 

4. Has the interface been properly constructed? 

Is the interface consistent ? 

Is help available when necessary? 

Can users easily navigate around the system? 

How does the user navigate around the system? 

How does the user know where s/he is? 

Is progression through the program logical? 

Can the user start and stop as required? 

5. Have the individual media been well-executed? 

Is the text /graphics, /sound etc well produced? 

Are the sections of text too long/too short? 

Will the text be understood by the target audience? 

Has text been expressed using elaboration, representation or abstraction? 
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This paper compares two grouping strategies for teaching computer literacy at 
the university. The authors and their colleagues have been involved in the 
computer literacy education to 180 freshmen majoring in Letters and 150 
freshmen majoring in Human Sciences for five years. In 1999, 180 students 
were organized into three classes at the early stage of the semester according to 
the student's wish, whereas 150 students were reorganized into three classes 
based on an achievement exam in the middle of the semester. The statistical 
analysis of exams and students' self-assessment showed that the early class 
organization was not so effective compared to the mid-term reorganization. 
The effect of mid-term reorganization was significant in the slow-leamers class. 

Keywords: class organization, computer literacy education, self-assessment, 
teaching strategy 



1 Introduction 

Computer literacy education in Japanese colleges and universities is facing a problem of diversity of 
students' computer skill levels. The reason for this is twofold. One is that very few high schools introduce 
the computer literacy education is into their curriculum. The other is that as computer price is falling down 
the number of students is increasing who have learned how to use word -processing or e-mails by themselves 
at home. This problem led us to study how to help our students learn this subject in good quality. 

In Osaka University, the computer literary education has been a required subject for all freshmen since 1994. 
The computer literacy course consists of 15 sessions, and each session lasts 90 minutes, which includes 
demonstrations by a teacher and hands-on activities by students. Usually an assignment for each session is 
also given which will take an hour or two to complete. Three teachers including the authors have been 
teaching the computer literacy course to over 150 freshmen majoring in human sciences, and our three 
colleagues have been involved in teaching to 1 80 students majoring in letters. 

In 1999, 150 students majoring in Human Sciences were organized into three parallel 50-student classes 
according to the enrollment order before the first session, h the middle of the semester, we gave an exam 
that includes hands-on work, such as word document processing, consulting online dictionaries, and finding 
web sites. Then we reorganized the three classes based on the exam score. Other background variables 
(age, gender and future academic field) were not considered for the class organization. The class 
organization aimed at ability grouping, that would provide non-experienced students with a slower work 
pace and allow high-achieving students to be sufficiently challenged by more demanding lessons. At the end 
of the semester, we again gave them an exam that would measure how they made progress. 

As for the 180 students majoring in Letters, three parallel 60-student classes were reorganized according to 
the students' wish based on the questionnaire survey at the fourth session of the course. That is, the 
keywords of the three new classes, “slow and steady”, “intermediate” and “intensive” were shown to 
students, and each student chose one of the classes. 
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In this paper, we begin with related work and a brief explanation of our computer literacy course, followed 
by the methodology of the class reorganization. Then, we present the statistical data analysis to show that 
class reorganization in the middle of the term was effective. 



#1 Change this plain text file to rich text format, then change the font of the marked words to Ryumin 12 
point, and place it in the center. 

#2 Explain how to use Japanese Kana-Kanji translation and how to register your name in your own 
dictionary. 

#3 Look up the three words (omitted here) in the following dictionaries: English-Japanese, 
Japanese-English, and Webster. 

#4 Draw a tiny picture and insert it into this page. 

#5 Find URLs of the National Diet Library and any two online book store s. 

#6 Look into "/SharedLibrary /Literature/" and describe what’s in it. 

#7 What should be considered when you decide your password string? 

#8 Find the specified string in the large file and copy the paragraph to the answer sheet. 

#9 Describe how to cope with chain mails. 

#10 Describe how to limit the line length to fewer than 65 characters in the body of an e-mail. 



Table 1 Midterm exam sample in 1999 



Touch typing 


(poor) 1 2 3 4 5 (excellent) 


Mouse operation (click, double click, drag) 


(poor) 1 2 3 4 5 (excellent) 


Window operation (resize, move, iconify, hide) 


(poor) 1 2 3 4 5 (excellent) 


Japanese Kana-Kanji translation 


(poor) 1 2 3 4 5 (excellent) 


File/Folder operation (move, rename, create) 


(poor) 1 2 3 4 5 (excellent) 


Word document processing (fonts, centering, insert graphics) 


(poor) 1 2 3 4 5 (excellent) 


Mail (MIME, signature, save to file, reply, delete, re -file) 


(poor) 1 2 3 4 5 (excellent) 


Web (search engine, book-mark) 


(poor) 1 2 3 4 5 (excellent) 


Use of online dictionary (Japanese, English) 


(poor) 12 3 4 5 (excellent) 


Canceling your printer job 


(poor) 1 2 3 4 5 (excellent) 



Table 2 Questionnaire survey for self-assessment 



2 Related work 

Although there have been much previous research on ability grouping, tracking, and class organization, they 
are mainly for K-12 school education, e.g. [4,9,10]. Furthermore, as far as the authors know, there has not 
been any research on class reorganization in the middle of a semester for computer literacy education. The 
authors have tried class reorganization for five years and presented its result obtained before 1998 in [5]. Our 
reorganization method is basically based on the exam score, but is not the same as cluster grouping. 



3 Computer literacy education 

Since the 1970s, the definition of computer literacy has evolved, and many researchers have discussed the 
courseware and teaching methodology of the computer literacy course, e.g. [2,3,4]. The authors believe that 
to learn details of word processing and spreadsheet application is not important but to grasp the concept and 
principal facilities of those applications is a key for students to become literate. Furthermore, the attitude to 
learn by himself or herself is also a key. 

In the latest syllabus of the computer literary course in Osaka University, topics consist of two categories, 
“ Requisites ” and “ Options ”, 

Requisite category consists of the following items. 

• File system, file operation, floppy disks 

• Word processing, kana-kanji conversion (for Japanese characters) 

• Concept of the Internet, network etiquette 

• Electronic mail, web 



• Draw and/or paint software 

• Spreadsheet 
Options are 

• Net news 

• LaTeX with graphics 

• Writing HTML 

• Mathematica 

• SAS (Statistics) 

• Script languages (perl, awk, shell, etc.) 

• Computer ethics, social problems 

Since one course session lasts 90 minutes and is held once a week, all topics of requisite category are the 
minimum competencies and are covered in about seven weeks. For the rest of one semester, teachers select 
some of the items from optional categoiy depending on the students' majors. For example, LaTeX and 
Mathematica are selected for students majoring in Physics and Mathematics, while spreadsheets and SAS for 
Economics. 



4 Class reorganization in the middle of a term 

In this section, we explain the class reorganization method that we adopted in 1999. Then we show the 
examination scores and self-assessment scores of the three classes. 

4.1 Reorganizing three classes by a hands-on performance exam 

After we completed topics in requisite category, we set a mid-term examination that demands 1-hour hands 
on work, because there were differences between students how fast they got accustomed to computer 
operations in class. Table 1 shows the questions of the mid-term exam. We marked the examination 
papers out of 100 (each question of 10), and used the scores to reorganize the classes. The students who 
achieved more than 67 points were grouped into class A. The threshold between class B (50 students) and 
class C (50 students) was 52 points. The average total score was 60.3 (s=15.7). 

The average scores of the new classes A, B, and C were 77 (s=7.4), 60 (s=3.9), 42 (s=7.5) points, 
respectively. The average score of class A students was more than 5.5 for all ten questions, whereas the 
students who were grouped into class C got only 2.2 and 2.3 for question #5 (web search) and #6 (file 
search), respectively, which were significantly lower than the other two classes. 

4.1.1 Self-assessment by students 

Before the mid-term exam, a questionnaire survey paper was distributed to the students. The survey 
consisted of 10 questions, which would measure students' mastery of competencies taught by that time 
(Table 2). 

The answers to these questions were collected on a 5-point scale. The scores of the collected questionnaire 
showed a positive correlation between the exam and the students' self-assessments. That is, class A 
students felt good at the following competencies: inserting graphics into a document, operation of 
files/folders, and searching web site, whereas class C students answered that they had not mastered those 
topics. As for searching files from hierarchical file structure, class C students perceived that they were 
good at it; however, its exam score is poor. This indicates that class C students did not understand the file 
structure itself. Touch-typing and Email competencies were not statistically significant between three 
classes. 

4.1.2 Curriculum after the class reorganization 

The result of the exam and the questionnaire showed us what we should teach in the new classes after the 
class reorganization. We provided class C students with a slower work pace and revisited those 
competencies that they were not good at. For class A students, we gave lectures more extensively than other 
two classes and gave self-teaching assignment of LaTeX in the summer vacation. 
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In order to allow them to have an active participation in the shaping and augmenting of their learning, we 
also introduced self-teaching approach for learning spreadsheets in all classes. That is, after a brief 
explanation given by a teacher, students followed tutorials on the teacher's web pages. We had two 
teaching assistants each class, we asked them not to teach detailed step-by-step operations to the students but 
to help the students to find the way by themselves. 

4.2 Evaluation of class reorganization 

4.2.1 Statistical evaluations of exam and questionnaire 

At the end of the course, we gave a self-assessment questionnaire survey followed by the term-end 
examination, which required hands-on work of about an hour. Here we begin with the analysis of the exam 
scores. 




midterm term-end 



Figure 1 Total scores of midterm and term-end exams 




Figure 2 Scores of self-assessment 

The total score and the scores of three questions (use of online dictionaries, OP AC search, and limiting line 
length of Mail body), which are similar in two exams, were considered in a two-way ANOVA. As a result, 
these four items showed an "interaction" between the two exams and classes (p < 0.05). 

Figure 1 shows a comparison of the average scores of two examinations. It indicates the existence of the 
"ceiling effect", that is, the score difference between class A and other two classes decreased after the class 
reorganization. 

As for spreadsheets, two sessions (which means 1 80 minutes) were assigned for class C, while one session 
was assigned for class A and B classes using self-learning web based text. But the scores were 8 points in 
class A and 5 points in class C. The score of writing HTML also supports this tendency. From this fact, it 
is proved that class A students had higher ability in computer operations than other two class students. 



4.2.2 Analysis of self-assessment questionnaire 

As for the 27 items that were included both mid-term and term end questionnaires, the average term end 
exam scores were higher than mid-term scores in all classes in 1998 and in 1999. In Figure 2, the x-axis 
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shows the midterm score, the y-axis shows the term-end score, and each scatter point in the graph represents 
the score of one item. Two years analysis showed same tendency. 

We took the exam timing (midterm or term-end) and the classes as the factors of the two-way analysis of 
variance (ANOVA). The following four items of the questionnaire shows that there is an “interaction” 
between the exam timing and the classes (p < 0.05). 

• Creating a new folder (directory) 

• Editing a word document 

• Browsing web pages 

• Searching web site 

We conclude that class C students perceived they were as self-confident in these items as the students in the 
other two classes. 

Though the exam scores are not significantly correlated with students' self-assessments, we consider the 
class reorganization was effective for slow learners class students, because it is important for them to have 
confidence in their computer operations. 

5 Comparison of class reorganizations 

There is another attempt to reorganize classes. The 180 students majoring in Letters chose one of three 
classes whose lecture policy are “intensive (class A)T, “intermediate (class B)” and “slow and steady (class 
C)”, Teachers expected that computer literate students might choose class A, and novice would choose class 
C, 

To evaluate two class organizations, we discuss the term-end exam scores and result of self-assessment. 

5.1 Comparison of the term-end exam score 

The students in six classes were given an examination at the end of the term. The full marks of each 
question were ten points. The following five questions are common to all classes. 

(a) Look a difficult Kanji word up in a Japanese online dictionary. 

(b) Explain how to cancel a printing job. 

(c) Find Yukio Mishima's book using online public access catalog of the university library. 

(d) Explain why 2-byte special characters should be avoided in e-mails. 

(e) Show URLs of the companies which sell cars on the web. 

In Figure 3, the square^ ) indicates the mean score of each class and the vertical line shows the 95% 
confidence interval of the mean score. To identify a particular class, we use H (for Human Sciences) and L 
(for Letters) as a subscript of the class name. For example. Ah denotes the“highest-score” class for students 
majoring in Human Sciences, Cl denotes the “slow-and-steady” class for students majoring in Letters. 

The result of the “one-way ANOVA ” and the “statistical multiple comparison” shows that questions (a), (b), 
(c) and (d) have statistical difference (p<0,05). Most of the confidence intervals of the class C are bigger 
than those of other classes, which means the variance of student competencies exists. 

The score of the question (b) of “H” classes is higher than “L” classes. This might due to the fact that the 
operation was not taught in“L” classes. 

5.2 Comparison of exam score and result of the self-assessment 

Before the term-end exam, the same questionnaire survey was also distributed to the L classes. Figure 4 
shows the distribution of self-assessment score of two items: consulting on-line dictionaries and canceling a 
printing job. The “one-way ANOVA” and the “statistical multiple comparison” show that the differences of 
mean scores among classes were significant in the Figure 4 (a) but not significant in the Figure 4 (b). 
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(a) Consulting on-line dictionaries 



(b) Canceling a printing job 





(c) Accessing on-line catalog of the library 



(d) Explaining special characters 



Figure 3 Distribution of the term-end exam scores 





(a) Consulting on-line dictionaries (b) Canceling a printing job 

Figure 4 Distribution of self-assessment scores 

The comparison between the exam scores and self-assessment shows that all of students who achieved high 
mark in the exam don’t necessarily perceive that they were very good at it. For example, although the 
scores are high in Figure 3 (a), the result of the self-assessment is not good. 

6 Conclusions 

In the first half of the paper, we described the effect of class reorganization in the middle of a term, which 
has enhanced learning outcomes in computer literacy classes. The class reorganization is also a good tool for 
teachers to know the way to lead students to higher skill level. 

Figure 1 and Figure 2 suggest that many students, especially for those who belonged to class C, have made 
progress during the latter half of the semester. The progress was brought by the two factors, that is, the 
students' internal motivation and the class reorganization. Moreover, the “interaction effect” of the two 
factors must be taken into account as a factorial effect. In order to measure the effect of two factors 
separately, an experiment based on ANOVA model is necessary. However, the score improvement of class C 
suggests the existence of the class reorganization effect. 
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In 2000, we will incorporate group work, discussion and presentation for high ability students after the class 
reorganization. 
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The web-based learning system “The Basic of Digital Media Communication” 
creates an interactive learning environment, which includes all the knowledge 
and aspects needed for the course. During the course, each student can choose 
freely either In Class Learning (ICL) or Web-Based Learning (WBL). Based on 
the questionnaires after the course, the paper evaluates the courseware system 
from the view of teaching and learning processes, interactive learning 
environments and learning effects. After the comparing study between ICL and 
WBL, it can be concluded that better computer and Internet environment can 
promote the students to accept the way of WBL, and WBL can enhance 
students* ability of self-study. Compared with ICL students, WBL students 
take less time and get higher score, and they are usually more efficient and 
individualized. The impact of traditional way of learning and interpersonal 
communication will exist for a long time, and there should be multi-way of 
learning to meet the different students. 

Keywords: Web-Based Learning, Evaluation, Interactive Learning 

Environments, Teaching and Learning Processes 

1 Introduction 

The course, ’’The Basic of Digital Medial Communication”, is a combination of computer application, 
communication and art design and it consists of basic theories, applications and creative design. That is, the 
basic knowledge of this course is about how to use the computer to organize, edit and deliver multimedia 
information. During the course, students need to do a lot of design works on computer. Because the content 
is so practical nowadays, the course is very popular among students. But since the course opened in 1996, it 
has confronted different problems concerning education mode and content. Firstly, the registration of the 
course is always limited by classroom and computer laboratory. Secondly, some of the reading materials 
were renewed and expanded nearly every year. Thirdly, because of the characteristics of the course, the most 
efficient way of learning and teaching is with the help of multimedia and network. Therefor, in the process 
of education reform, the course has experienced three ways of teaching and learning mode, that is, 
traditional learning, multimedia -assisted learning and Web-Based Learning. While improving the course 
gradually, a web-based courseware • The Basic of Digital Medial Communication* have been developed(ij. 
This courseware system constructs a comprehensive and interactive learning environment. The data capacity 
of the courseware is about 1 Gega Byte totally, and it includes all information and aspects needed for the 
course, such as the schedule, learning materials, experiment instruction, reference materials, demonstration 
and examples, answering questions, homework handed in and feedback, discussions and so on. 

The object of the courseware is to construct a new learning mode, which should inspire the students’ 
creativity and innovation. While instructed by teacher and assisted with modem education technologies, the 
students should be the center in the teaching and learning process. The course is carry on in an open and 
interactive environment, and the main features are as following: 

(1) Open Computer Laboratories of the University are opened for the students, and they can visit the 
courseware at anywhere and on anytime if they logon to Internet. All the information about the course is 
published on the courseware, and the latest news or notice is always renewed according to teaching and 
learning process. The students can ask questions and discuss them with others through the courseware. 




(2) According to the schedule, the students should finish a serial of media design work step by step. All the 
works can be handed in through web. Not long after hand in, each student can look up the web and get the 
score and comment from the teacher for each design work. The excellent works and the teacher’ s 
comments will be published on the web on time, and the students can refer to and discuss about the works. 

(3) In class lectures are based on the courseware, which will be projected on multimedia classroom. 
Discussing classes are arranged during the semester, which are also based on the courseware and projector. 
In this paper, we call this way of learning as In Class Learning (ICL). 

(4) Except the ICL, the students can also choose the way of Web-Based Learning (WBL). In this way, the 
students need not attend the in class lectures, but they should finished all the design work and final test by 
self-leaning through the courseware system. The requirement of design work and final test are the same 
for both ICL and WBL students. 

At first, the courseware system was used in a course composed of 30 students. Then, in the fall of 1999, it 
was used in a selective course opening to all the undergraduates and each student can freely choose the way 
of ICL or WBL, While the design work requirements and the test for both way are the same. At the end of 
the semester, two kinds of questionnaires were carried out among two group chose different learning modes. 
The questionnaires were designed in three aspects: the courseware system, the teaching and learning process 
and the learning effect. Compared and analyzed the feedback from the two groups, it can be concluded that, 
the web-based courseware system and learning mode have reached expected objects, and the learning effects 
have connection to the computer environment, the self-study ability of students and the impacts of 
traditional way of learning. 

2 Web-Based Learning is the trend of modern education development 

At the beginning of the semester, only 25% of the total 70 students chose the way of WBL. The others either 
chose ICL or couldn’ t make their mind yet. After a period of time, when they got more familiar with the 
courseware system, all the students uncertain and 18% of those chose ICL originally turned to WBL. 
Consequently, the WBL students have made up of 61% of the total students registered. According to the 
investigation, the students’ choice could be affected mainly by three factors, that is, self-study ability, 
computer and Internet environment and the basic knowledge for the course. 

Among all the students, 72% of the juniors and seniors chose WBL, while only 46% of the freshmen and 
sophomores chose WBL. Those who chose the WBL consider it a better way, because the place, time and 
learning schedule can all been controlled freely by themselves. This in turn requires the students to be of 
higher ability of self-study. Most juniors and seniors prefer WBL, for they are usually better at self-study. 
From this point of view, higher ability of self-study is needed for WBL, and WBL will improve their ability 
of self-study, which is also the basic ability for life-long learning. 

In order to visit the courseware without the limitation of time and space, Internet logon is the basic 
environment for the students. There are about 500 computers connected to Internet in the Open Laboratory 
of our University, but it’ s always full of the students. More and more students have their own computers in 
dormitories, and more and more dormitories are connected to the Internet via high speed cable. From the 
investigation, most students chose WBL have better computer and Internet environment than those chose 
ICL. Therefor, the better the computer & Internet environment, the more the students who will accept WBL. 

Because of the course’ s feature, computer is both the tool and object of learning the course. Compared to 
ICL group, most WBL students consider they had basic idea about the course at the beginning of the 
semester. That means that WBL students have basic ability of computer application, which could been 
called as digital literacy and it’ s the basic literacy demand in the information age pj. So, the digital literacy is 
another factors that can affect their choices. In the process of WBL, computers and web are basic tools of 
learning, and it will definitely improve one’ s ability for computer application or digital literacy. 

3 Web-Based Learning is of more efficiency and individuality 

WBL has more advantages over traditional way of learning. WBL has changed the traditional relationship 
between teacher and students, where the teacher is the initiative disseminator and the students are passive 
receivers. The courseware enables the students to study individually. They can decide how long to visit the 
courseware each time and which part to concentrate on. What’ s more, they can do the homework or design 
work at the same time while reading the courseware, which is more efficient than traditional way of in class 
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lecture and then design work. 



Take the learning procedure of one chapter in the course for an example. In the teaching schedule, 6 in class 
hours are planed to finish the chapter and the relative design work. Analyzing the percentage of students and 
the total hours used to finish this chapter* s learning and design work (Fig 1), the statistic result shows that 
the hours used by WBL students tend to be more changeable. The average hours used by WBL and ICL are 
6.5 and 8 hours respectively, while the average scores of this design work are 77.7 and 79.8 respectively. 
That is, WBL group turns to be take less average time and get higher marks. 




parentage of 
CL studnets 

percentage of 
WL students 



Fig 1 : Percentage of students and the total hours used to learn chapter 7 

Study the total time used for the course (Fig 2), it also shows that the ICL students tend to arrange their 
learning time according to teacher’ s schedule, while WBL students tend to arrange their time individually. 
From the statistic of how WBL students read the courseware, only about 35% of them learn systematically 
and about 45% of them consult the courseware only if they need help. But WBL students’ average total 
score is 84, higher than 80 of ICL students. Although scores are just one way to test the effect of learning, 
from all statistic results and analysis, it shows that WBL is of more efficiency and individuality. 




percentage of 
CL students 

percentage of 
WL students 



Fig 2: Percentage of students and the total time used for the course 
(note: T is the total in class time of the course’ s schedule) 

4 The courseware system plays an important part in the learning process 

The courseware is both the learning object and an effective learning tool, so it is very important in the 
learning process. At the beginning of the semester, teacher announces the schedule, homework requirement 
on the courseware. Then the Students can learn the knowledge from the courseware step by step according 
to the schedule, hand in their homework, ask questions and discuss relevant topics through the courseware. 
The answers to the common questions, the score and comment of everyone’ s homework are also published 
on the courseware. The latest news on the courseware is renewed at least two times a week. Though parts of 
the learning materials are almost the same with the textbook, the courseware is more attractive because its 
multimedia, multiform and interaction. 

The courseware is the main learning materials for the two groups, and more than one third of the students 
completed the learning only by the courseware. The top successful aspects of the course from the two group are 
showed in Table 1. When designing the courseware system, it was not expected that homework hand in and 
feedback by the courseware would be so popular among the students. The students whose excellent design 
work is published on the courseware are very proud about it, and the others would be greatly inspired and 
they will make their efforts to improve their own works too. So in the courseware, not only the database of 
learning material but also the interactive environment has played very important part in the teaching and 
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learning process. Both WBL and ICL students evaluate the course highly, and the courseware system has 
reached its expected objects. 





ICL 


WBL 


1 


Get the latest news from the courseware 


Web-based learning 


2 


Homework hand in and feedback via the courseware 


Homework hand in and feedback via the courseware 




Discussing class by projector 


Get the latest news from the courseware 


4 


In class lecture by projector 





Table 1 : Statistic of the top successful aspects of the course 



5 There should be multi-way of learning to meet the different students 

At the end of the semester, most students feel satisfaction to the learning mode they chose, no matter WBL 
or ICL. Meanwhile, they all have good evaluations to self-study abilities. The two groups have different 
options about “discussing class”. 68% of the ICL students think the discussion is indispensable and should 
be emphasized, for they can have chance to communicate with others in person, while 21% of the WBL 
students think it’ s unnecessary, for they can “discuss” in the courseware. It’ s clearly showed that, ICL 
students tend to be more rely on the interpersonal communication, while WBL students tend to be more rely 
on network communication. The weakening of interpersonal communication is a new problem emerging in 
the information age, and it will influence learning effect^. 

Analyzed which is “the best way of learning in college”, 27% of all the students accept WBL completely, 
while 58% consider the best way is combining WBL and ICL, that is what we have done in the course. This 
kind of combination is suitable for students with different basis. For those tend to WBL, they can learn 
individually by the courseware system and consult the teacher in discussing class when they want. For those 
tend to ICL, they can get systematical instruction from the teacher in classroom, and use the courseware 
resource after class. In a long term, coexistence of multi-way of learning is a practical solution, which will 
instruct the learning and motivate creativity of students simultaneously. 

6 Conclusion 

Web-Based Learning has many advantages over traditional way, while it will take a long period of time for it 
to be perfected. Web-Based Learning has not only changed the teaching and learning process, but also the 
education mode and teaching thoughts. Its success depends on the network hardware, courseware and the 
efforts of the teacher and the students. In the course “The Basic of Digital Media Communication”, the 
students are so eager for knowledge and so interesting in the educational reform. In fact, the courseware 
system is developed and improved during the teaching and learning process. 

Web-Based Learning is the trend of education in today’ s information age. This kind of learning is different 
from comp letely self-study. Though the interactive learning environments of the courseware system, the 
teacher instructs the students how to learn, and encourages them to study independently. Although the 
content is professional, the structure and learning mode of the courseware system have common sense for 
other courses. In order to satisfy different students and enhance their ability as well, the in class lectures will 
be decreased gradually, but discussing classes will be remained both for interpersonal communication and 
for answering the question face to face. In order to cultivate the higher qualified students in the information 
age, the education mode should stress the ability of acquiring knowledge and self-learning. This in turn, 
requires the teacher to be higher qualified too. 
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Computer mediated collaborative projects have the potential to strengthen the 
European Dimension in teacher education whilst giving students an appropriate 
context to develop their computing and collaborative skills. This paper evaluates 
the success of such a project through the completion of a three-year action 
research enquiry involving student teachers from four European countries. The 
results of three cycles of development are presented. The project was evaluated 
using student questionnaire data, participation in tutor meetings, and analysis of 
students’ web page development and bulletin board contributions. Results 
suggest that successful collaborative project work depends on ease of access to 
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reliable computer networks, giving equal weighting to resource production and 
levels of international communication, and effective moderation of the project 
by all tutors involved. The paper concludes by detailing future developments in 
European cooperation involving the partner institutions. These developments 
involve using the Ecoschool communication networks to discuss pedagogic and 
multi-media design issues involved in a cross-curricular CD-ROM which has 
been developed by the same group of partner institutions. 

Keywords: computer mediated communication, European co-operation, 
moderation. 



1 Introduction 

This paper reviews a three-year cycle of telematics curriculum development and action-research in initial 
teacher education. The project has been made possible by funding via the SOCRATES European Module 
ECOSCHOOL (1997-2000). The project has two aims; to develop learning by using the World Wide Web 
(WWW) and email across Europe, and to learn about the social and economic aspects of the participant* s 
home city. The outcomes of the project include the creation of a collaborative open learning course that 
teacher education students can follow as part of their training. 

The Ecoschool developments originated from European collaboration on the EUROLAND project 
(1996-99). It brings together partners from Austria, England, Finland and the Netherlands in building the 
European Dimension into the curriculum of schools and teacher education courses (Hudson et al, 1997 and 
Hudson et al, 1999). Teacher education institutions and departments lead both projects in close collaboration 
with partner schools and teachers in each country. The resources that have been produced by both the 
Ecoschool and Euroland projects have been used as the basis for the development of pedagogic approaches 
with teachers on intensive in-service training courses, which have been supported under the Comenius 3.2 
Action of SOCRATES. 

The paper reports on four aspects of the Ecoschool project; the three year cycle of curriculum development, 
the tutor and student evaluation of the project, lessons learned regarding telematics pedagogy, and future 
developments that link the outcomes of the Euroland and Ecoschool projects. 

1.1 Participants in the project 



The participants are primary teacher education students from Linz and Sheffield together with students on an 
international teacher education course at Oulu. A more recent partner to this development is the University 
of Darlana in Sweden. This has led to the participation of a group of social studies student teachers from 
Falun in Sweden. English was used as the medium for communication and a total of eighty five students 
took part over the three years. 

1.2 Collaboration and communication 

A key aim of the project has been to promote the European Dimension and the use of Information and 
Communications Technology (ICT) in teacher education across Europe. The development of the European 
Dimension provides ample justification for collaborative communication but such projects can also reflect 
sound pedagogic principles. The pedagogical approach is based on a socio -cultural communicative 
perspective, which owes much to the works of Vygotsky (1987). Collaborative learning is at the heart of the 
Ecoschool project and has been used during the three cycles of student work. Many authors, including 
Hudson (1998) and English and Yazdani (1999) see such an approach as essential in developing students' 
learning skills when using ICT or learning without the aid of new technology. 



2 Use of new technologies 

The resources and tools being used are university email communications and the resources provided by the 
ProTo environment at the University of Oulu- Project Learning Tools on the Web. This is an open learning 
environment that has been developed at the University of Oulu. Students can access the ProTo system via 
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the World Wide Web. They have a password that allows them to create simple web pages and enter 
messages on a bulletin board. Students also created web pages using Netscape Composer and posted them 
on their home pages. In cycle three they used an electronic bulletin board as well as using ProTo and 
email. 

Use of such technology is now a key focus in the education of teachers across Europe. Student teachers in 
England and Wales follow the National Curriculum for Initial Teacher Training (DfEE 1998). This 
curriculum requires students to show evidence of using and creating multi-media presentations, and of using 
web technologies to communicate with colleagues. In addition, recently published guidance detailing an ICT 
primary school curriculum (QCA, 1998), suggests that children aged ten should be able to design and 
evaluate simple multi-media presentations, and children aged eight should be able to take part in an email 
exchange. Clearly student teachers need the confidence and skills to develop these abilities in their pupils. 
The Ecoschool gives students this experience through their participation in a computer mediated 
collaborative project and by their evaluation of its potential use in their future educational roles. 

2.1 Pedagogic approaches 

As previously stated the Ecoschool project uses a pedagogic approach that seeks to promote learning 
through 'electronic talk' in collaborative groups. These groups use a plan, do and review strategy as proposed 
by Kolb (1984) in his model of experiential education and by Schon (1987 ) describing the planning cycle 
used by reflective teachers and learners. The groups planned the construction of webpages, constructed and 
evaluated their own pages and those of other groups, then finally evaluated the whole project. Tutors 
developed their own pedagogy of distance learning during the project. The success of the tutors' approaches 
are analysed using guidance developed by McGee and Boyd (1995) to facilitate dialogue during computer 
mediated communication. 



3 The three cycles of curriculum development 

3.1 Cycle One 

Focus : comparing students' home cities 
Outcomes: web pages explaining local city 

Figure 1 : Work from the Swedish students posted to the 
ProTo learning environment. 

Students in each country worked in 
collaborative groups to produce a short 
illustrated report on one of the following 
aspects of their home city. This 
involved a general description of the city, 
an explanation of the environmental 
situation and the employment structure of 
the city, and an analysis of the regional or 
national education 
system. 

Subsequently they presented these 
reports as web pages by writing them in 
to the ProTo learning environment. 
Figure 1 shows a page produced by the 
Swedish students. They also emailed 
their work to other students in the partner 
countries who were presenting the same 
topic. Once all web pages were complete, they read their partner’ s pages, asked questions and made 
comments about them on the bulletin board. Each group evaluated their work using the same criteria 
designed by the tutors in each country. The tutors then read each group’ s pages, assessed the pages and 
provided feedback to the each group. The students’ work was assessed against the criteria and graded A to C. 
The tutors posted written feedback on the bulletin board. 
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3.2 Cycle Two 



Focus : Comparing lesson planning 

Outcomes: web pages giving examples of lesson plans 

Figure 2: Teaching and learning about the environment in Lin 
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3.3 Cycle Three 



Focus: suggesting and solving educational problems 
Outcomes: range of solutions to five educational problems 



Figure 3 The Euroland and Ecoschool discussion and chat site. 
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The aim of this round of co-operation 
was for students to share lesson plans and 
teaching ideas. Each group of students 
planned lessons with the aim of children 
learning more about their local town or 
city. Again, students presented these as 
web pages on the ProTo system or, in the 
case of the Swedish students, on their 
university home pages. Each group of 
students again evaluated the pages of 
their partner groups, responded to each 
other's questions, and received feedback 
ffom the tutors in each country. Students' 
work was again assessed. 



The Ecoschool project ran during autumn 
1999 with several new developments. The 
students were in internationally composed 
groups rather than from one single country 
and the focus of the project was to choose 
an educational problem and present a 
solution to this by co-operating using ICT. 
The students could use email, create their 
own web pages, use ProTo2 (a more 
sophisticated version), or use the 
Ecoschool bulletin board (see Figure 3). 
The majority of students chose to use the 
bulletin board to present their problem and 
solutions although some students did use 
the ProTo2 learning environment. Again 
tutors gave feedback to the students and 
responded to their questions although the 
work was not graded. 



4 Methods of curriculum development and evaluation 

Ecoschool developments have followed an action research model, as the aim of the project was to develop a 
successful curriculum for initial teacher education over the three years of the project. The Ecoschool 
curriculum was developed in face-to-face planning meetings and followed up by email communication 
between partners in Austria, Finland, Sweden and England. The results of student and tutor evaluation were 
fed into the curriculum planning at the end of each cycle. The following methods have been employed in 
gathering evaluation data: 
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Student evaluation questionnaire. All students completed a questionnaire by email or on paper. Many 
groups posted the results of their evaluation on the ProTo system or on the Ecoschool bulletin board. The 
questionnaire requested information on student expectations of the project, levels of interaction, the role of 
the tutor, use of new technology and ideas for the future. 

Tutor evaluation. A tutor from each country completed a written evaluation of their experience at the end of 
each cycle and presented the document for discussion at the annual Ecoschool development meetings. 

Web page analysis. The students created web pages of differing levels of complexity during cycles one and 
two. The web pages construction process is evaluated against the six components of infomedia literacy as 
proposed by Lee (1999, pp. 147-149). These components are: 

1 . An understanding of the nature and functions of infomedia and their impact on individuals and society. 

2. The development of critical thinking ability. 

3. The skill of efficient searching and critical selection of information. 

4. Knowledge of multi-media production using appropriate technology. 

5. Aesthetic appreciation of hypertext, graphic design and visual images. 

6. Social participation in influencing the development of infomedia technology. 

ProTo communication log analysis. The record of tutor and student communication during cycles one and 
two was analysed using Boyd and McGee's (1995) guidance on facilitating dialogues using 
computer- mediated communication. They suggest that facilitators provide both technical and 
content-specific support; are responsible for regularly communicating with the group; communicate in ways 
that require a response; and model standards of high quality interaction 

Ecoschool bulletin board observation. The Ecoschool bulletin board was set up in September 1999 and 
provided the student groups with a shared electronic space for presenting and discussing their ideas. Each 
group had a separate area for their own use. The frequency and quality of communication was analysed as 
well as the level of interaction between group members. 



5 Evaluation Results 

Student evaluation questionnaire data was collected from 12 of the 16 student groups over the three years. 

The key points arising were: 

• In cycles one and two students who were apprehensive about using the technology felt that had been 
successful and the majority of students found that resource production was enjoyable and had developed 
their ICT skills. 

• Communication between groups was successful in cycles one and two but sporadic in cycle three. This 
was attributed to pressure of work from other areas of their degree (Oulu), lack of clarity in terms of the 
aims of the project and technological problems in Linz and Sheffield. 

• In cycle three, two of the five groups were critical of the lack of commitment of their partners. 

• Students in Sheffield requested formal computer sessions where they could meet and use university 
facilities for the project. All students felt that their tutors had supported them in cycles one and two, but 
three groups wanted clearer guidance in cycle three. 

• By cycle three the students from Falun and Oulu requested the use of chat and video conferencing 
technology in any future work. Individual students in Linz and Sheffield experienced technical 
difficulties during November 1999 due to network problems at their institutions. 

Minutes of three Tutor evaluation meetings and five written reports state that: 

• The role of the tutor was clear in cycles one and two but not in cycle three 

• Cycle three was seen as a radical departure from previous work and was viewed as 'experimental' 

• Students in Linz, Oulu and Sheffield were hampered by block teaching practices taking place during 
key times in the project. 

• Tutors were pleased with the progress made by their students in cycles one and two and had discussed 
how work in cycle three could be improved. 

Web page analysis using Lee's concept of infomedia literacy reveals: 

• Only two groups took a critical approach to the sources they used when constructing pages about their 
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home city in cycle one. 

• Three groups overtly discussed the problems of representing people and places on their web pages in 
cycle two. 

• Four groups of students in cycle one saw the pages as similar to written text so did not exploit the 
advantages of hypertext fully. 

• All students changed from passive users of web pages to active publishers of their own content. 

• The students from Falun produced a website in cycle two that clearly demonstrated a collaborative 
approach and a high level of aesthetic appreciation in regard to page design. 

• Students from Oulu and Falun were in general more adept at making critical comments about their own 
and other's work than the Linz and Sheffield students. 

ProTo communication log and Ecoschool bulletin board observation using McGee and Young's guidance 

shows: 

• In cycles one and two tutors adequately fulfilled the roles of moderator, mediator and facilitator. 

• Tutors communicated with the participants by asking one or more questions, giving examples from their 
own experience to add to discussions and modelling high quality interaction. 

• The cycle three work led to the production of questions and solutions but little discussion. In general 
tutors did not moderate the discussion effectively as they were unsure of their roles. 

• The decision to limit the role of the tutor in cycle three had a negative effect on the level of interaction 
and quality of discussion. Student evaluations reveal uncertainty about technical issues as well as 
pleas for stronger leadership and rigid deadlines. 



6 Discussion 

The cycles of curriculum development and evaluation have identified many important features in the 
development of collaborative ICT projects. Establishing an ntemational electronic community requires 
access to reliable technology for the students and also skill and commitment on the part of the tutors. 
Asynchronous communication is seen as one of the great advantages of electronic communication and 
university tutors may take their own ease of access for granted. In a study of barriers to student computer 
usage McMahon et al (1999) found that students identify real problems in accessing computers to complete 
course tasks. A Sheffield student reflects their conclusions when evaluating her experience in cycle three: 

If we had been given time in our lectures to get together and a set routine with 
correspondence time every week then we would have got more out of it. As a group of 
people we are all in different (teaching) groups, so getting together is difficult and finding 
a PC when we have free time is also difficult. 

This highlights the question of computer access as well as the importance of study and group work skills in 
such a project. Very clear project goals and explicit expectations on student participation are also needed. Is 
it the students ' responsibility to meet and organise communication sessions during their own time, or will 
better levels of communication occur by booking computer access during student practical classes? If this is 
done are the benefits of asynchronous communication being demonstrated? An unexpected outcome of this 
project has been to highlight the importance of developing students' teamworking skills. 

Once access is assured, the roles of the tutor as moderator, mediator and facilitator are crucial. A key finding 
from the evaluation is that communication was much more successful when the tutors had a strong 
moderating role in cycles one and two. When planning for cycle three, tutors limited the moderation role and 
gave the student groups much more independence. The majority of the students interpreted this as poor 
planning and one group described it as 'lack of leadership'. This highlights the complexity of the moderator's 
role and a recommendation from this project would be that the tutors spend time in the final evaluation 
meeting exploring their experiences in this role. 

Developing a successful collaborative curriculum is dependent on creating a fine balance between resource 
production and communication. In cycle one the web pages produced were basic, but quality of interaction 
between students was high. In cycle two the web-based products were much more sophisticated but students 
paid less attention to communication, perhaps because more academic credit was gained for page 
development rather than communicating with fellow students. Student's work in the final cycle showed 
some evidence of sound international cooperation, but less in-depth critical analysis. Experiences gained 
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during the three cycles have led to the development of a formal curriculum unit (see 
http://www.shu.ac.uk/schools/ed/teaching/dhoA . Students will gain high grades only by giving equal 
weighting to communication, resource production and critical evaluation in their group work. 

Finally, teacher education students need to transfer their learning to a classroom situation. One student has 
already set up a similar project whilst on teaching practice. In this example infant school children 
communicated v;a email with children in Bermuda and compared their localities, hobbies and homes as part 
of English and geography learning. Tutors need to set up opportunities for students to use their newfound 
confidence and skill in the classroom. Nook Lane Primary School in Sheffield is now linked with partner 
schools in Linz and Oulu as a result of the project, and students can now contribute to the development of 
this partnership. 

6.1 Future developments 

As a result of ongoing evaluation the following developments have been planned for 2000-2001. A chat 
facility had been added to the Euroland/Ecoschool discussion area in addition to the bulletin boards. 
Building on the success of a trial video-conferencing session held in November 1999, students will be able 
to use this form of communication from September 2000 in all countries. Students and teachers can also 
now access the communication tools via the Hallam Geography Education web site as well as from the 
Euroland web pages. Finally, with the imminent completion of the Euroland CD-ROM, the two projects will 
be brought together. Students and teachers will be able to use the CD-ROM as a focus for collaboration and 
discussion in the areas of infomedia literacy and multi-media development, the pedagogy of 
computer-mediated collaboration and the comparison of European social and environmental learning. 

7 Conclusion 

The Euroland and Ecoschool projects represent successful examples of how an international perspective can 
be developed in the university and school curricula. Sustained and effective communication is the key to 
such initiatives, alongside ease of access to computing facilities and a focus on the crucial role of the tutor as 
moderator. Both projects have provided tutors, students and pupils with membership of an expanding 
European network, which is a solid platform for the development of further collaborative work. 
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In this paper we discuss our observations of a group of 10 and 1 1 year old children 
using an Interactive Learning Environment called the Ecolab. The design of this 
software was informed by our interpretation of Vygotsky s Zone of Proximal 
Development in which Interaction and Collaboration are definitive characteristics. 
The relationship between the differences in interaction/collaboration style and the 
learning gains made by the children are discussed. The results show that children 
can be grouped into profiles according to the differences and similarities in their use 
of the system and that common interaction features are influenced by the design of 
the software being used. We suggest that children are poor at managing their own 
learning experience with technology even when the software offers both 
opportunities to complete challenging activities and support to ensure success. The 
children in this study needed explicit direction towards activities which were beyond 
their ability. However, caution with regard to this provision of direction is 
important to ensure that the child is also offered opportunities for creativity: a 
suggestion from the system about what and how to proceed is often sufficient. 

Keywords: Interaction, Collaboration, ZPD, ELE. 



1 Introduction 

Computers are now an accepted part of classroom life for most young learners whether they are used for 
communication, visualization, simulation experience or simply for fun. But how do children actually interact 
with computers? Does the nature of their interactions vary from child to child in a way that could inform the 
design of the software which engenders these interactions? This paper explores children s use of an Interactive 
Learning Environment (ILE) called the Ecolab which was designed to help children learn about ecology. The 
system attempts to fulfill the role of a more able learning partner for the child and invites collaborative 
interaction. The collaboration is thus between the system and the child and not between children. Here we 
describe the nature of the interactions that a class of children had with this system. The nature of these 
interactions is considered in the light of pre- and post-test learning gains to explore the relationship between 
learning and interaction style. The Ecolab software has been designed using a framework derived from our 
interpretations of the Zone of Proximal Development (ZPD) [10, 11]. The ZPD describes the most fertile 
interactions which occur between the more and less able members of an educational culture and focuses attention 
on how the more able can help learners to learn. The ZPD offers a theory of instruction which emphasizes the 
inseparability of the teaching and learning processes and thus recognizes the inherent interactivity of children s 
learning with computer software. It also stresses the need for learners to have the help of a collaborative 
learning partner in the form of a peer, a teacher or in the case of the Ecolab, a computer. Within a Vygotskian, 
socio-cultural model of education human activity is mediated by tools and sign systems that have arisen 
through social interaction. Developmental explanations are used to address the complex internalisation process 
by which the interpsychological relations between partners in social interaction becomes intrapsycho logical 
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within the individual learner. Interaction and Collaboration are therefore definitive characteristics of the ZPD 
which form the linchpin of the socio-cultural framework and thus form the focus of our investigations of 
children using the software. 

In this paper we provide a brief description of the Ecolab software before discussing an evaluation study of its 
use. We report the results with particular emphasis upon the nature of the Interaction and Collaboration profiles 
we were able to construct from our records of system use. We provide examples of individual learner s use of 
the system and discuss the relationship between the nature of the interactions and the learning gains recorded 
after system use. 



2 Ecolab Software 

Ecology is a subject that involves the study of relationships between organisms within our environment. These 
relationships can be extremely complex; they can also be introduced in a simplified manner through concepts 
such as food chains and food webs. These form the foundations of more complex ecosystems and are part of 
the curriculum for primary school children in the United Kingdom. The Ecolab software provides 10 and 1 1 
year old children with the facilities to build, activate and observe the ecological relationships which exist 
between members of a simple food web in a woodland ecosystem. It provides a simulated ecology laboratory 
environment into which the child places the animals and plants of her choice. This environment can be viewed 
by the child from several different perspectives or views, including: 

World - a picture of a woodland environment and the organisms the child has chosen to place within it. 

Web - a traditional text book style diagram of the organisms in a food chain and food web. 

Energy - a graphical representation of the energy levels of the organisms currently alive in the Ecolab. 

History - a linear narrative of what has happened in the Ecolab world to date, which animal has eaten which 
other animal for example. 

As we have already stated the nature of the relationships that can exist between organisms in the real world can 
be very complex. We wished to allow each of the children using our system to learn about relationships at a 
level of complexity that was appropriate to them. We therefore built the learning environment in a manner that 
would allow children to learn about relationships ranging from the simplest, between just two single organisms, 
to the much more complex network of relationships that could exist in a very simple ecosystem involving 
populations of organisms. The complexity of the relationships represented within the Ecolab can be varied at 
any stage during the child’s interaction with it. It is also possible to alter the abstractness of the terminology 
used to describe the organisms in the Ecolab so that a snail, for example, can be described by the words 
“herbivore”, “primary consumer”, or “consumer” as well as the word “snail”. 

In addition to this simulated laboratory environment, the system acts as a collaborative learning partner for each 
learner which can provide assistance of the following sorts: 

Extension of the learner s knowledge through increasing the complexity of the relationships she is asked to 
study and/or the abstractness of the terminology used to describe what is happening in the Ecolab. 

Collaborative Support which can take the shape of Activity Differentiation: in the form of alterations to the 

difficulty of the activities the learner is asked to complete, or context sensitive Help of variable levels of quality 
and quantity. 

At the start of this paper we discussed our use of the Zone of Proximal Development to underpin our system 
design and the importance of Interaction and Collaboration. In order to explore the nature of the interactions 
children had with our software, the collaboration that might occur between system and learner, and the 
relationship between interaction, collaboration and the changes in learning outcome recorded after system use, 
we varied the manner in which collaboration from the system was offered to the learner. The Ecolab consists 
of three system variations: VIS (Vygotskian Inspired System), WIS (Woodsian Inspired System) and NIS (Non- 
theoretically Inspired System). These three system manipulations implement different design elements in order 
to adjust the assistance they provide (see [4] and [5] for more detail). The way in which each of the system 
variations adopts a different approach is summarised in Table 1. In particular, VIS makes more decisions than 
WIS which makes more decisions than NIS. In other words NIS gives the learner most freedom of choice to 
the learner and VIS the least. 
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3 Interactions with the Ecolab 



An exploratory evaluation of the Ecolab software was conducted with a class of children aged 1 0 and 1 1 years. 
We wanted to investigate the extent to which the system would be able to adjust to learners of differing 

abilities, and also the ways in which the interactions and collaborations between user and system varied with 

users of different abilities. The children’s school assessments were therefore used to allocate each child to one 
of three ability grouping: High, Average and Low. Prior to using the software each child completed a written 
and a verbal pre-test, the latter of which was in the form of a structured interview recorded on audio tape. Each 
child used the Ecolab software as an individual for a total of 60 minutes over two sessions. In addition, a 20 
minute initial session with a smaller ’demo’ version ensured that all children were comfortable with the mouse 
skills required and the interface. After the system intervention subjects were given a written and verbal test, 
identical to the pre-test, and a short additional extension interview. A delayed post-test was conducted 10 
weeks after the end of the original post-test. Of the 30 children who started the study only 26 completed all 
sessions between, and including, pre and post-test. The four who did not complete these sessions had either 

left the school or been absent during the evaluation period. Only 24 completed all sessions including the 

delayed post- test. Once again the reason for non-completion was absence from school. 

Table 1 Collaborative Support within Ecolab 



Collaborative Support within Ecolab 




VIS 


WIS 


NIS 


Levels of Help Available 
(different levels provide 
differing qualities of help 
- 5 represents the greatest 
and 1 the least) 


5 


5 


2 


Decision about Level of 
Help made by 


system 


system and child 


child 


Levels of Activity 

Differentiation Available 


3 


3 


3 


Decision about type of 
Activity and 

Differentiation level 

made by 


system 


child - system makes 
suggestions 


child 


Extent of Learner Model 
maintained by the 

system and used to make 
decisions about the 
support to be offered to 
the learner. 


Bayesian Belief Network 
(BBN) of values 

representing the system’s 
beliefs about child’s ZPD 
formed from its 

knowledge about the 
amount of collaborative 
support used to date. 


Record of help used to 
enable contingent 

calculation of next help 
level. Record of 

curriculum nodes visited 
maintained to permit 
suggestions. 


Record of Curriculum 
nodes visited maintained 
to help child keep track. 


Abstractness of 

Terminology selected by 


system 


child 


child 


Area of the Curriculum 
and complexity of the 
next activity selected by 


system 


child - system makes 
suggestions 


child 


Ecolab View selected by 


mostly child 


child 


child 



The results of the pre- and post-test were used to assess the efficacy of the three variations of the Ecolab 
software. This work is reported elsewhere [4, 5] and is not the main focus of the current paper. It is the 
character of the interactions between each child and the system that we will focus upon here. We wanted to 
investigate what sorts of interactions had resulted in the greater learning gains and which systems had supported 
and encouraged various types of interaction and collaboration in order to inform the design of our next system. 
For each child a summary record of their interactions was produced from the detailed logs maintained during 
their two sessions of system use and this was used to build up a picture of the types of interactions each child 
experienced with the system (for full information see [4]). 

Cognitive or learning styles have been a subject of active interest in recent years [1, 3, 6, 8], for a brief review 
see [9]. The influence which a learner’s style can have upon the way they interact with technology has also been 
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recognised [7]. Within this literature there are examples of classification systems which differentiate learners 
according to their learning preferences; for example, as serialists or holists [6]. The analysis of the annotated 
interaction summaries of children’s experiences with the Ecolab software takes a fresh perspective on 

classification using only the styles of interaction or Profiles which can be found in the records of each child’s 

system use and emphasizing our interest in the nature of Interaction and Collaboration . Characteristics were 
identified and children categorised through: 

• Interaction Profiles according to the character of their interactions with the Ecolab. 

• Collaboration Profiles according to the nature of the collaborative support provided by the system for the 

child. 



4 Results 

One aspect of the evaluation looked at whether the different variations of the Ecolab had been more or less 
effective in increasing the child’s learning gain in terms of her understanding of the feeding relationships which 
exist in a food web as reflected in the pre- and post-test data. This indicated that the system variation (VIS, 
WIS or NIS) which the child used was relevant to her subsequent learning gain and a detailed discussion of 
these results can be found in [5]. Here we wish to concentrate upon the analysis of the records of interaction 
which was used to try and pinpoint the elements of VIS and WIS which led to their superior performance with 
particular ability groups. 

4.1 Interaction profiles. 

There were two characteristics which could clearly be seen as either present, or largely absent within the 
children s interactions. These were referred to as: 

• Busyness and 

• Exploration 

Busyness was considered to be a characteristic of interactions in which the children completed an average or 
above average number of actions of any type, such as adding an organism to their Ecolab world or making one 
organism eat another. The interaction summaries of these children contained an above average number of 
events. The opposite of Busyness is referred to as Quietness. 

Exploration was considered to be a characteristic of an interaction if the child had been involved in some sort of 
action which allowed her to experience more than one level of complexity or more than one level of 
terminology abstraction, beyond her initial starting levels. The opposite of Exploration is referred to as 
Consolidation. 

Some children also switched frequently from one type of interaction to another. For example, they might 
switch from attempting to make one animal eat another, to looking at their organisms in a different view (i.e. 
perspective), to accessing a new activity entirely. Their interactions contained no or few series of repeated 
actions of the same type. They were particularly prone to frequent changes of view. These users have been 
characterised as hoppers. Other learners exhibited a more persistent approach, with sets of actions of a similar 
type grouped together. These users have been referred to as persisters. 

These characteristics allow the children to be categorised , in principle, into 1 of 8 (2x2x2) possible Interaction 
Profiles. 

The three parameters of categorisation: Busy/Quiet, Exploration/Consolidation and Hopper/Per sister bear some 
similarity to features found in other categorisation systems. Pask’s [6] differentiation of tendencies in learners 
towards being either "top-down" holists or being "bottom-up" serialists shares some common ground with the 
Hopper/P ersister characteristic, for example. The differentiation of exploration from continuing activity at a 
level of consolidation is likewise similar to the challenge/safety division of [2]. However, the motivation for 
the analysis reported in this paper was not the presentation of a generally applicable categorisation system. The 
aim was twofold: 
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• To investigate the relationship between interaction style and learning gain. 

• To examine how each of the system variations (VIS, WIS and NIS) of the Ecolab supported and 
encouraged particular learning styles. 

Children fell into 6 of the 8 possible Interaction Profile groups. The distribution within these groups is 
illustrated in Table 2. 



Table 2 Interaction Profile Membership 



Profile Description 


% of children in Profile group 


Busy - Exploring - Persister (BEP) 


28% 


Busy - Exploring - Hopper (BEH) 


12% 


Busy - Consolidating - Persister (BCP) 


8% 


Busy - Consolidating - Hopper (BCH) 


12% 


Quiet - Consolidating - Persister (QCP) 


20% 


Quiet - Exploring - Persister (QEP) 


20% 



4.1.1 Examples of User Interaction Profiles 

S10 (Gene) was a typical example of the Busy - Exploring - Persister style of interacter. Her first action was to 
switch from world view to energy view and then back to world view. She then added 1 5 organisms to the 
Ecolab and visited energy view again. Upon switching back to world view she made one of her organisms eat 
another, switching to energy view to see the effect. This pattern of making organisms act, either eating or 
moving and looking at the effect in an increasing number of different views continued. Introductory, 
investigative and rule-definition activity types were completed for the first two nodes in the curriculum before 
her first session drew to a close. She chose not to save her current Ecolab world which meant that at the start 
of her next session her first actions were the addition of organisms. Once again she added all 15 and then 
moved into the next phase of food web complexity and used more abstract terminology to view her organisms. 
Whilst the nature of the actions she completed was now more advanced and several instances of help were used, 
her pattern of activity remained one of initiating an action or actions appropriate to the evident goal. Actions 
were often completed in pairs and were followed by viewing the result from different perspectives (most 
commonly, energy, web and world). She did not experiment with writing a program or attempt to escape 
from completing the activities offered to her. 

This profile group contains only high and average ability children from the VIS and WIS system user groups. 
In terms of performance at post-test there was a tremendous spread: A Busy - Exploring - Persister style learner 
attained the lowest learning gain, another, the second highest learning gain. The high ability children within 
the group all achieved an above average learning gain, but within the average ability children there was a wider 
spread of learning gain scores. Membership of this group was limited to VIS and WIS users, of whom the VIS 
users all achieved above average post-test learning gains, including the highest learning gain within this user 
group. 

4.2 Collaboration profiles. 

Two characteristics were found to be the most useful for differentiating collaborative style within the 
interactions: Amount of support and Depth of support used. These collaboration characteristics were used to 
group the children into one of four Collaboration Profile groups. 

Amount of support: the average amount of activity differentiation (i.e. the degree to which the activity is 
presented in a simpler form) and the average number of help instances for the experimental group was 
calculated. An above average amount of either activity differentiation or instances of help was the criteria 
necessary for a child to be considered as using ’Lots’ of collaborative support. 

Depth of support: this characteristic was based upon the level of help and level of differentiation used. Once 
again the average levels used within the experimental group were calculated. Help or differentiation above the 
average level resulted in a child being considered as using ’Deep’ or higher level support. 
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Interactions could be grouped into all 4 of the possible Collaboration Profiles. The first group was the largest 
and was further divided in accordance with the type of support which was most prevalent. The distribution of 
children into these groups is illustrated in Table 3. 

Table 3 Distribution of children within Collaboration Profile groups 



Profile Description 


% of children 
in Profile 


Profile sub-group 
Description 


% of children 
in Profile sub- 
-group 


Lots and Deep (LD) 


53% 


Differentiation 
and Help 


19% 






Differentiation 


19% 






Help 


15% 


Lots and Shallow (LND) 


12% 






Little and Deep (NLD) 


16% 






Little and Shallow (NLND) 


19% 







4.2.1 Examples of User Collaboration Profiles 

SI (Jason’s) use of the available support was typical of the Lots and Deep profile group and of a user of above 
average amounts of both help and activity differentiation. He used level 4 help early in his first session of 
system use to achieve success in making organisms eat each other. His initial activities were completed with 
maximum differentiation of level 3. This was gradually reduced and then increased again. During his first 
session of system use he completed a range of activities for three nodes in the first phase of the curriculum. All 
instances of successful help were at level 4 or level 5. Fewer activities were completed during his second 
session. However, these activities were at a lower level of differentiation and there were fewer instances of 
help. 

This Collaboration Profile group was the largest and was subdivided to account for the type of support used. 
Only VIS and WIS system users shared the profile. Jason was a member of the subgroup which used above 
average amounts and levels of both activity differentiation and help. This subgroup again consisted only of 
high and average ability children whose mean learning gain is above the average for the whole class (16% as 
compared to the class average of 1 1.5%). The subgroup of children who used greater levels of differentiation 
than help contained children from all ability groups. This second subgroup also produced above average 
learning gains at post-test (18% as compared to the class average of 1 1.5%). The last subgroup of children, 
who used greater amounts of help than differentiation, were all average ability children. Their average learning 
gain was well below the class average (3.9% as compared to the class average of 1 1 .5%). 

System variation had a greater impact upon the nature of the Interaction and Collaboration profiles than ability. 
A Pearson Chi -squared statistical test was also used to assess the relationship between the Ability groups, 
System Variation Groups, Interaction Profile Groups and the Collaboration Profile Groups. There was a 
significant association between System variation membership and Collaboration Profile membership (X z = 
28.52, df = 6, p < .0001), and also between System variation membership and Interaction Profile membership 
(X 2 = 25.79, df = 10, p < .01). 

So far little has been said about the NIS user group, they have not belonged to either of the Profiles used in the 
examples. In fact, all the NIS users belonged to a Consolidating Interaction profile; there were no explorers in 
this system user group. In addition, and as has previously been mentioned, no NIS users were in the Lots and 
Deep Collaboration profile group. 

S9’s (Tim’s) Interaction profile which was that of a Quiet, Consolidating Persister, was typical of a NIS system 
user. His initial session consisted of adding a single snail and then making 1 1 view changes to look at this 
organism from all perspectives. This initial stage was followed by a series of organism additions (commonly 
in blocks of 4); single actions, such as move or eat commands, in blocks of 1 to 5; and view changes which 
were almost always in pairs. In session 2 he adopted the commonly seen approach of adding a considerable 
number of organisms to start (in this case 12) and then once again completing single actions and view changes. 



Likewise S26 (Karlie) s Collaboration profile reflecting low use of all types of help (Little and Shallow: 
NLND) was typical. She placed herself at the far extreme of food web complexity and started dealing with 
populations of organisms straight away. She only completed one type of action during both sessions of 
computer use: she built food webs using the build web command. Initially she made errors and used only 
occasional low level feedback, persisting until successful. The children in this profile group were all of high or 
average ability, but their average learning gains were well below average (5.2% as compared to the class average 
of 11.5%) 

A further difference found within the NIS user group relates to the relationship between ability and learning 
gain. In the VIS and WIS user groups it was the higher ability children who achieved the greatest learning 
gains. By contrast, amongst the NIS users none of the high ability children made an above average learning 
gain, in fact the only learners who made above average learning gains were the low ability children. Whilst the 
numbers are small and the study exploratory this result is interesting and is certainly informing our current 
research. We had expected that of all three systems, the one which left most control within the hands of the 
learner would be most effective with the more able learners. Our results indicate that the opposite was in fact 
the case in our study. 



5 Conclusions 

This is an initial exploratory study with small numbers of children. However, there are several observations 
which are informative in building up a picture of the sorts of interactions which children experienced with the 
version of the system they used. VIS was the system which explicitly selected the next curriculum area for the 
child to complete and controlled the complexity and abstractness of the learning environment. Not 
surprisingly, all VIS users were members of profile groups with the 'Exploring' characteristic present. The split 
between 'Busy' and 'Quiet' was almost even. Only two of the VIS users scored a below averagdeaming gain at 
post-test and both were in the same 'Quiet, Exploring, Persisted profile group. The majority ofWIS users were 
also 'Exploring' profile group members and only one did not belong to a 'Busy' profile group. However, whilst 
all the WIS above average learning gain achievers were members of 'Exploring' profile groups, the below 
average achievers were all members of different profile groups, with no common features between all of them. 
The WIS system variation did not set the curriculum area for the users, but did make suggestions which 
resulted in it being easier for a WIS user to avoid being an 'Explorer' thana VIS user. The NIS users were the 
children with the greatest freedom and the least finely tuned help system. It is perhaps not surprising therefore 
that none of them belonged to a profile group with the 'Exploring' characteristic. They were evenly split 
between being 'Busy' and 'Quiet' and the majority were 'Persisters'. Only two NIS users achievedbove average 
learning gains and unlike the WIS and VIS users, both were in profile groups which shared the 'Comfortable' 
characteristic, they were also both in the low ability group. 

These results suggest that simply providing children with the means for extension through becoming involved 
in challenging activities is not enough to ensure that these challenging activities are undertaken. The child 
needs also to be explicitly directed towards activities which are beyond her ability. However, caution with 
regard to this provision of direction is important to ensure that the child is also offered opportunities for 
creativity. The success of WIS indicates that a suggestion about what and how to proceed is often sufficient. 
The consistency within the high and average ability groups across the different systems for above average 
learning gain achievement to be linked to the 'Exploring' profile characteristic isnot reflected in the low ability 
group. The definition of the 'Exploring' characteristic may of coursebe too crude to encompass the possibility 
that the low ability children were 'Exploring' within interactions in a single phase of the Ecolab. 

The manner in which each variation of the system collaborates with the child is a design feature of that variation 
and as such a big influence upon the resultant user Collaboration Profile. It was no surprise, therefore, that 
there was a significant association between system variation and collaborative support profile membership. 
However, it is possible, in principle, for a user of any of the variations to interact in line with any of the 
Collaboration Profiles described. In reality Collaboration Profile 'Lots and Deep’ was exclusive to VIS and 
WIS users, whereas Collaboration Profiles 'Lots and Shallow’ and'Little and Deep’ were exclusive to WIS and 
NIS users. The only system which allocated both help and differentiation to users was VIS, so the fact that 
VIS users all used a high quantity and quality of help is unsurprising. WIS users often used a high level of 
assistance too, but in smaller quantities, they all belong to profiles where the support used was of a high level. 
In contrast, all NIS users are in profile groups in which the level of support is low. The choice of help 
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available to NIS users was admittedly more limited being of only two levels, however none of the users ever 
chose to use the higher level of help offered. 

The absence of some forms of assistance from the interaction summaries of the less successful users offers 
support for the suggestion that it is the combination of being challenged, or extended, plus the provision of 
ample quantities and qualities of support which is important for learning. The lower ability children present a 
somewhat different picture as there is no apparent consistency between the use of collaborative support and 
learning gain. The only tentative conclusions are that this group responded to interactions in which the extent 
of the challenge was limited and that the nature of the assistance the system could offer was not effective for 
them. Those who were successful took up less different types of assistance and tackled less of the curriculum 
than their successful more able peers. There is also evidence that these children were not good at managing 
their own learning. The NIS Interaction and Collaboration profiles in particular would suggest that children 
who are given control for their own learning experience are not good at setting themselves challenging tasks or 
indeed seeking collaborative support. Our current work with children is investigating this issue in more depth. 
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Today, computer literacy courses in general education are common in Japanese 
universities. Usually, such courses include word processing and spread sheet 
calculation, in which the control operation is given by clicking a mouse cursor 
on a computer display. Students tend to pay attention only to the mouse and 
the display, which are situated at the most external level of the computer. Thus, 
it is very difficult for many students to understand what is going on in the 
internal level of the computer and the present status of learning. The above point 
of view has led us to the following hypothesis: Observing the movement of data 
and control of the operating system in the internal level helps students 
understand the essence, which will be a basis of their further studies on 
computers. We have designed and implemented two kinds of computer 
simulators: (1) The one that visualizes the internal execution of a computer, and 
(2) the one that does not visualize the internal execution. We made experimental 
uses of both simulators in an actual course and compared the two results. We 
concluded that the simulator visualizing internal execution has a better effect. 
Furthermore, we obtained the following result: Presenting the movement of the 
data and control of the operating system helps students understand the computer 
operation. 

Keywords: computer literacy, internal execution, computer operation, 
visualization 



1 Introduction 

As the availability of computers increase, more computer literacy education is widely offered at universities, 
including those departments affiliated with the liberal arts. The instruction offered usually includes how to 
use word processors, spreadsheets , and how to access the Internet. The operating system used in most cases 
is Microsoft Windows and the most commonly used input device is a mouse. 

It is often observed that students have a perfunctory and vague attitude when responding to their 
manipulation of fragmentary data appearing on the screen with a mouse. However, if students are able to 
comprehend what is taking place inside the computer and also have a general idea of the state of the OS 
(operating system) at each moment, they will be able to understand the fundamentals of the computer 
operation. This, in turn, would make subsequent learning easier. In other words, the potential of further 
understanding the computer is limited when only superficial operating procedures are learnt. Therefore, we 
hypothesized that visualizing the computer's internal operations would contribute to the student's higher 
understanding of the process, such as how a file is moved to a storage location by the current manipulation, 
which is one step beyond the computer display. In 'The Report on General Information Processing 
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Educational) published by the Information Processing Society of Japan, the importance of computer 
literacy education is mentioned. However, many universities are struggling to come up with effective 
teaching methods. (2), (3) 

In the past, there were researches done (4), (5) in education that used computers to simulate the computer 
itself, for example. Further researches on dynamically visualizing the contents of learning include an 
algorithm animation (6),(7),(8),(9),(10), a program execution and visualization of calculation operations 
(1 1), (12), and visualization of OS (13), (14). Most of these researches are concerned with students of 
computer science, whereas this paper takes up an internal simulation of Windows OS, which is frequently 
used in the general computer literacy education including that offered to liberal arts students. In this study, a 
system was developed that simulates computer operations that goes one step deeper than the normal 
computer "exterior". The authors then carried out an experiment using the system to a group of liberal arts 
students and evaluated it. 

The purpose of this study is to increase the learner’s understanding and subsequent learning by providing 
visualization of the permanent storage, file movement, and the change in OS controls within the computer. 



2 Development of the Simulator 

2.1 Structure of the Simulator 

The computer system used in the simulator includes a computer, display, keyboard, printer, and a floppy disk. 
Two locations inside the computer are shown schematically; The one is a hard disk, which we will be calling 
a "permanent storage" in the following sections, where files and software are stored; the other one is a 
memory area, which we will be calling a "temporary storage", used as working space, where files and 
software retrieved from the permanent storage are transferred to. The OS used was Windows and the word 
processing software Microsoft Word was simulated. Visualization of two word processing operations 
involving, 

(1) change in the OS control 

(2) file (data) transfer 

were devised, and two types of simulators were developed. 

The one, which we call type A in the following sections, visualizes the computer's internal permanent and 
temporary storages along with the previously mentioned computer equipment, and simulates the movement 
of the Windows OS controls and the file transfer The other one, which we call type B, simulates only the 
Windows OS controls and file transfer operations; in other words, the internal parts of the computer is not 
included in type B. 

Furthermore, the learner’s use of the simulator can be recorded to show a learning history. Visual Basic Ver. 
5 was used as the language for developing the simulators. 

2.2 Simulator Functions 

2.2.1 Type A Simulator Functions 

When the simulator starts, the same environment is presented as when the learner uses MS Word. The 
input-output devices as well as the computer’s internal permanent and temporary storage locations that 
cannot be seen from the outside are displayed (see Figure 1). Although the data in the permanent storage are 
maintained even if the power supply is turned off, the data in the temporary storage are lost. 

The simulator has a color display in order to make it easier to understand visually. Initially, the entire 
background is in gray and only the computer display, like the real one, is in black. The color red is used for 
important iems. When the power is turned on, the computer display turns into a light cyan color and 
becomes ready for operation. At each stage, significant colors are assigned to regions where files and 
software such as Windows and Word are allocated. The colors are assigned based on the operations and the 
timing when such operations occur When the permanent and temporary storage areas become ready for use, 
the background changes to light blue, and when the software or files are executed, that region's color 
changes to yellow, and then changes to orange the next time something is added to it. When quitting, 
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although the area returns to the gray color, if there is text in the memory, the text is shown in black to 
indicate that it remains as part of the memory. 

In addition, an arrow (->) is employed to show the controls relating to Windows and Word, and another 
arrow (->) is used to show the file and data transfer. 

In the simulator, the internal operations of the computer are reproduced according to the state transitions, 
and expressed according to the commands contained in Word’s File menu. The names of the software, 
devices and areas are those of the simulator used in the simulator’s screen display. 

(1) Before turning on the power, the student enters his/her student identification number and name in the 
spaces on the screen in response to prompts for the purpose of recording the learning history. 

(2) Power is turned on. 

A red power switch (alternates between ”on” and ’’off’) located at the bottom right-hand comer of the 
display is pushed. Windows starts up from files stored in the permanent storage and the Word icon is 
displayed at the bottom of the display. 

(3) Starting Word 

Word is started using the Word icon. The permanent storage, temporary storage, and the display regions 
for Word are set up. [File (F)] for the file menu appears in the upper left comer of the screen. An 
arrow is displayed to indicate that the file transfer and the text editing operations are under Wordk 
control. 

(4) The work is proceeded by selecting the following prescribed menus from the "File" menu. 

• New Document (N) 

Text is input and shown in two yellow areas indicating the temporary storage and the display. An 
example is shown in Figure 1 . 

• Open(O) 

The name of the saved document is input into the dialog box that appears on the screen. The saved file 
is displayed in the temporary storage and the display. 

• Close (C) 

The display and the temporary storage are closed. The text of the document located in the permanent 
storage is displayed. 

• Save (S) 

The text is added to the document in real time. It is saved in the permanent storage in the same location 
as the original document. 

• Save As (A) 

When an arbitrary file name is input into the dialog box presented on the screen, the document in the 
temporary storage is transferred to the permanent storage. An example is shown in Figure 2. 

• Print (P) 

The document in the temporary storage is shown for printing. 

• Quit (X) 

Word is shut down and the text in the temporary storage and the display disappears. The contents of the 
permanent storage remain, and the region turns into gray. 




Figure 1 Example of the "New Document" menu Figure 2 Example ofthe "Save As" menu 
by simulator A by simulator A 
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(5) Quitting Word 

To quit Word, the Quit (X) command from the File menu is selected. The learning history during the 
use of the simulator is saved. 

(6) Power is Turned Off 

The power is turned off when the power switch is pressed. 



2.2.2 Type B Simulator Functions 



The display of simulator B does not show the internal parts of the computer including the permanent and 
temporary storages unlike the previous simulator. Consequently, only the display, keyboard, printer, and the 
floppy disk that are seen during a normal operation of the computer are displayed. All the simulator 
functions related to these devices are exactly the same as the simulator A. An example of the "Print" menu is 
shown in Figure 3. 




Figure 3 Example of the "print" menu 
by simulator B 




Figure 4 Usage of simulator 



3 Experiment Using the Simulator 



3.1 Method of the Experiment 

Experiments were conducted using the two simulators, A and B. Subjects included 77 students 
(ranging from freshmen to seniors) from the law department of a private university. The subjects had already 
learned a word processing application (Word) and how to access the Internet in the first half of the semester 
in a beginner’s computer course. Subjects were divided into two groups, with 39 subjects using the 
simulator A and 38 subjects using the simulator B. Both groups answered questionnaires prior to their 
participation, performed a variety of tasks using the simulators, and answered questionnaires at the end of 
the study. Contents of the questionnaires were the same for both simulator A and simulator B groups. The 
initial questionnaire asked about the subject’s personal characteristics and 16 questions on the use of Word. 
The post-study questionnaire included the same 16 questions and additional ones related to their final 
impressions of using the simulator. The subjects were asked to select one of four answers (correct, probably 
correct, probably incorrect, and incorrect) in order to assess the subject's degree of confidence vhen 
answering. 

3.2 Description of the Experiment 



The following three tasks were given to both simulator A and simulator B groups during usage: 

( 1 ) Turn on the power, start Word, input text into a new document, and save the document as a named file. 

(2) Open the saved file and print it. 

(3) Add text to the saved file and save. Turn off the power. 

These tasks were conducted in sequence and performed once without exception. The second time subjects 
used the simulator, they were permitted to perform any of the available functions they found interesting 
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without limitation. The usage of simulator is shown in Figure 4. 



4 Results of the Experiment 

4.1 Analysis of Variance of Results 

Scores are analyzed by analysis of variance. The sources of variance included (1) method: the two 
methods employed in simulator A and simulator B respectively, (2) time: pre- and post-simulator usage, and 
(3) questions: the 16 questions in the questionnaire. 

The answers to the questions were weighted according to the following scheme: Correct (4 points), probably 
correct (3 points), probably incorrect (2 points), and incorrect (1 point). The analysis of variance is shown in 
Table 1. 

A significant difference of 1% level of significance(p<0.01) was observed between the 
methods used for simulator A and Simulator B. A significant difference of 1% level of 
significance(p<0.01) was also observed for time and for questions respectively. There was 
also interaction of 1% level of significance(p<0.01) between time and the questions. However, 
there was no significant interaction between the remaining combinations, that is, between method and time, 
method and questions, or method, time and questions. 



Table 1 Analysis of variance in simulator usage experiment 



Source 


DF 


Sum of 
Squares 


Mean 

Square 


F Value 


Pr > F 


Method 

(type A .type B ) 


1 


5.769 


5.769 


10.30 


0.0013 


Time 

(pre.post) 


1 


14.344 


14.344 


25.62 


0.0001 


Question 


15 


312.882 


20.859 


37.26 


0.0001 


Time*Question 


15 


23.786 


1.586 


2.83 


0.0002 


Method*Time 


1 


0.355 


0.355 


0.63 


0.4262 


Method*Question 


15 


5.274 


0.352 


0.63 


0.8541 


Mthd*Time*Qstn 


15 


6.294 


0.420 


0.75 


0.7350 


Error 


2400 


1343.71 


0.560 







♦♦p<0.01 



An analysis of variance of pre- and post- usage of simulator A and simulator B was conducted using the 
mean of each subject's total scores. The results are shown in Table 2. 

A significant difference of 5% (p<0.05) was observed with regards to pre- and post- usage of simulator A 
suggesting that the simulator experience had a significant effect. The pre - and post- usage of simulator B 
also showed a difference of less than 10% significant tendency (p<0.10). 

In contrast, no significant difference was observed between pre -usage of simulators A and B, nor between 
post-usage of simulators A and B. 



Table 2 Analysis of variance of pre- and post- usage of simulator 





pre 


post 


type A 


54.744 


57.564 


type B 


53.579 


55.632 



(Mean of Tatal score) 



type A : Test of the difference between pre- and post- usage 





DF 


Sum of 
Squares 


Mean 

Square 


F Value 


Pr> F 


Time 


1 


155.13 


155.13 


5.25 


0.0247 


Error 


76 


2245.03 


29.54 






Total 


77 


2400.15 









* p<0.05 

type B : Test of the difference between pre- and post- usage 





DF 


Sum of 
Squares 


Mean 

Square 


F Value 


Pr > F 


Time 


1 


80.05 


80.05 


3.32 


0.0726 


Error 


74 


1786.1 1 


24.14 






Total 


75 


1866.16 









4.2 Discussion of Analysis of Variance of the Experiment 

As mentioned in 4.1, a significant difference of 1% (p<0.01) was seen between simulators A and B. In 
addition, a significant difference of 5% (p<0.05) was seen between the pre- and post-usage of simulator A. 
Thus, it is regarded that the learners' understanding increased as a result of visually experiencing the 
computer's internal processes and the movement of files with simulator A. Understanding how files actually 
move inside a computer in relation to the operations they have already learned may have contributed to 
clearing leameis' ambiguous understanding about how computers handle files. We can therefore conclude 
that visualizing the internal operations of the computer by simulator A was effective. 

On the other hand, there was also a significant tendency in the results of the use of simulator B. The fact 
that exercising conventional operations in a similar operational environment still had an effective tendency 
suggests that the use of the simulator helped them acquire a condensed or simplified perception of how the 
files are manipulated. This could also be understood from the impression gained after analyzing the post- 
simulator usage questionnaires. 



5 Changes in student's understanding for each problem 



(1) "Windows" goes into operation at start-up. 

(2) "Word" is under "Windows" management. 

(3) Text that is not saved will disappear when the computer is shut down. 

(4) Saved files can be retrieved again. 

(5) "Word" goes into operation by double-clicking an icon. 

(6) The document stored inside the computer disappears when it is printed. 

(7) In order to protect against lightening, long text should sometimes be saved while inputting. 

(8) When a file is closed, the text displayed on the screen or in temporary storage does not disappear. 

(9) When a file is opened, it is shown on the display or called to the temporary storage. 

(10) When a file is closed, the saved document disappear. 

(1 1) When a file is saved, it is contained in the storage(floppy disk or hard disk). 

(12) If a blank screen is saved, the file is empty. 

(13) When you quit "Word", the text is displaying on the screen. 

(14) When you open a file and change the content, then it is impossible to save again. 

(15) You should specify a name when retrieving a file. 

( 1 6) * 1 1 is not guaranteed that the document is saved when the system or software quits suddenly.* 

Table 3 Summary of Questionnaire 
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The problems were 16 questions relating to the use of "Word". A simple summary is shown in Table 3. 

An analysis of variance was conducted with respect to the pre- and post- usage of simulator A. The results 
are shown in Table 4. 

The results of the F-test indicated changes before and after the simulator usage in the following situations. 

(1) . For simulator A, answers to Q2 (Word is controlled by Windows) showed a significant difference of 1% 
(p<0.01), Q13 (Quitting Word) also showed a significant difference of 5% (p<0.05), and Q5 (Word icon) 
showed less than 10% significant tendency of less than 10%. The post-usage effect of simulator A on 
questions related to the OS was quite apparent. Use of simulator A is thus thought to be successful since the 
students were able to grasp the general notion that the application being used was operating under the 
control of the OS. In particular, correct answers to Q2 increased as much as 60% with respect to the before 
and after ratio using simulator A, and this increase is far greater than that for simulator B (34%). This result 
shows that displaying the internal operations of the computer system was highly effective. 

(2) Q9 (Opening a saved file) and Q1 1 (Saving a file) involve retrieving data from and storing data in the 
permanent storage which are basic file manipulation operations. In simulator A, the permanent storage 
and transfer operations are visualized, and showed a significant effect with a significant difference of 1% 
(p<0.0 1 ) seen for both Q9 and Q 1 1 . 

Table 4 Analysis of variance of each question 
pre- and post- usage of simulator A 



Question 


pre -type A 
Mean 


post-typeA 

Mean 


F Value 


Pr > F 




1 


3.92 


3.92 


0.00 


1.0000 




2 


2.95 


3.79 


37.55 


0.0001 


** 


3 


3.56 


3.56 


0.00 


1.0000 




4 


3.90 


3.95 


0.71 


0.4020 




5 


3.21 


3.56 


2.90 


0.0929 


t 


6 


3.74 


3.77 


0.07 


0.7952 




7 


3.28 


3.51 


1.99 


0.1623 




8 


2.56 


2.46 


0.17 


0.6838 




9 


3.23 


3.69 


7.30 


0.0085 


** 


10 


3.54 


3.62 


0.21 


0.6448 




11 


3.51 


3.87 


7.32 


0.0084 


** 


12 


3.00 


3.05 


0.04 


0.8408 




13 


3.26 


3.62 


4.28 


0.0421 


* 


14 


3.79 


3.74 


0.20 


0.6585 




15 


3.82 


3.90 


0.77 


0.3818 




16 


3.46 


3.54 


0.28 


0.5986 





number of samples : 39 
** • p<0.01 

* • p<0.05 

t : p<0. 10 significant tendency 



6 Conclusions 

Two types of simulators that simulate the operation of Word under the Windows OS were developed for the 
computer literacy education. Operation tests involving 77 university students were performed using both 
types of simulators. The results indicated that simulator A, which simulated the internal operations of the 
computer, was more effective. 

We believe that when learning how to operate a computer, learning only the operating procedures makes it 
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difficult to understand the status of the computer at any given moment. 

Therefore, in order to help students understand what is going on in the present situation of learning, we 
developed a simulator that visualized internal executions. The results of the experiment using simulator A, 
which showed the internal executions, and simulator B, which did not, revealed the following. 

(1) A significant difference of 1% (p<0.01) was observed between simulator A, which displayed the 
computer's internal executions, and simulator B, which did not. A significant difference of 5% (p<0.05) 
was seen between the pre- and post- usage of simulator A. It indicates that simulator A was more 
effective. 

(2) Using simulator A, internal operations such as OS control, retrieving data from the permanent storage 
area, and saving data to the permanent storage area were well understood. 

These results indicate that visualization of the internal executions of a computer is effective in computer 
literacy education. 
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Nowadays, a large amount of digital images are being stored worldwide in 
Internet. As an educational means, images stored in Internet have a big potential, 
Teacheis can show the students pictures or images instead of actual things. The 
Internet is so rapidly expanding and becomes so complicated that the ways to 
retrieve images effectively from Internet database are now getting more difficult, 
In this paper we consider an image database about metadata type. We place a 
special emphasis on keyword itself in the metadata, and show the criteria of 
keyword, not the framework of them. Good keywords are needed in the 
database so that the retriever can get what he/she really wants. First we survey 
the necessity of metadata: especially keyword for image database. Next we 
consider and present the criteria for consistent and appropriate keywords, 
distinguishing subjective keywords from objective ones. And we examine and 
assess them. Furthermore we append a new item, importance, to our criteria. 

Keywords* image Retrieval, Metadata, Criteria 



1 Introduction 

Over the years large amounts of computer-aided images are being stored in Internet owing to widely 
available digital recording devices, such as digital cameras, scanners, and economical large size storage. For 
the effective management of those digital images, an image album or an image filing system has the subject 
of study and it has remarkably developed. And the Internet is so rapidly expanding and becomes so 
complicated that the effective retrieval methods of images from database becomes more and more important. 

As an educational means, image presentation is very important. Because teacheis cannot usually bring actual 
thing into classrooms, they show students the pictures or images instead. Or when the students use the 
Internet by themselves the images surely help them to learn fast and effectively. 

There are 3 kinds of image database: feature type, sensitiveness type, and metadata type. Feature type (cited 
as [1] and [2]) is based on the colors or shapes of object in the images. When the retriever puts in color or 
shape, the system starts searching directly the database with color histograms or shape. In the case of 
sensitiveness type (cited as [3] and [4]), the retriever puts in sensitive words and the system exchanges them 
for color information and searches the database. In the case of metadata type (cited as [5]), each image in the 
database is already given metadata, which explains its characteristics by texts or digits, and the retriever 
searches the database using the metadata. 

Our concern in this paper is of metadata type. Of this type, first, database creators define the structure or the 
framework of metadata. Second, database administrators attach metadata to images according to it in the 
database. Third, a retriever specifies texts as a key to the database. Finally the database system searches 
images using the metadata which is given by administrator and also using the texts which are keyed in by the 
retriever. Examples of metadata are keywords, texts, classification items and so on. 

The metadata of image database system metadata needs consistency and appropriateness. If it often happens 



122 



that some of metadata are irregular or incomplete, the retriever cannot find images which he/she really wants. 
Especially of commercial systems, the reputation of the database is determined by the quality of metadata. 

We place special emphasis on keywords in the metadata because they are the basic component of metadata. 
In Section 2 the necessity of metadata, especially keyword of image database, is discussed. In Section 3 the 
criteria of consistent and appropriate keywords is considered. In Section 4 the criterion, which we discussed 
in Section 3, is examined. In Section 5 our conclusion is presented and the future work is discussed. 

By the way, we are not concerned here the structure or the framework of metadata. It needs another 
consideration. In ISO, the structure of metadata for multimedia contents description, MPEG-7 [6], is now 
being standardized. MPEG-7 will provide the distribution and utilization of multimedia contents with 
content-based retrieval. The application will be distance learning, a stream database, a personal TV, and so 
on. MPEG-7 will become International Standard at September 200 1 . 



2 Metadata and Keyword 

Good metadata is needed in the database so that the retriever can get what he/she really wants. In storing 
database with images, we give texts, especially words as metadata to them. These words can be defined as 
keywords. They are given to images and used when a retriever searches the database. Database system 
searches images based on keywords which he/she specifies. 

Therefore when incompatible keywords are given to images in the database, a retriever cannot get images 
which he/she really wants, even if he/she puts in any compatible keywords. In next Section we show the 
criteria of good keyword, which discussed and experienced before in [7]. 



3 The criteria of keyword 

We shall discuss the criteria of keyword in detail. 

• Giving appropriate keywords 

A retriever depends only on keywords. Therefore the images need to be given beforehand the keywords 
which represent them precisely. For example, we should distinguish “people” from “doll” because he/she 
may really want the image of a doll, not of people. Or it depends on the retrievers of the database which is 
appropriate “car” or “BMW”. The database administrator should expect how the retriever uses keywords 
when he/she gives keywords to the image. 

• As many keywords as possible 

An image has a lot of views. For an example, in Figure* • some people takes it for the figure of pond, 
some of others for monument. Obviously only a single keyword is not adequate to describe its whole 
concept. 

Therefore, a nurrber of keywords should be given to an image so that it can precisely correspond to the 
keywords. The retriever can more easily obtain the images which he/she really wants. 

• Distinguish subjective keywords from objective 

We can define a lot of keywords for one image. They may be divided into two types. One is objective; the 
other is subjective. In figure* • objective keywords are, for example, “pond”, and “monument”. Whereas 
“summer 7 ’ and “shine” are subjective factors, because they are viewef s impressions of this image. They are 
subjective keywords. The former is more general than the latter. 

You might think that you need only objective keywords. However, a retrieved s needs are generally divided 
into two types: subjective and objective. In Figure* • he/she wants a summer picture. “Summer 7 ’ is a 
subjective word. If the database does not have subjective keyword, he/she cannot find such an image. He/she 
must hit objective keywords instead of “summer 7 ’. Accordingly there are subjective needs in so many cases 
that subjective keywords are to be needed. For this reason, both subjective and objective keywords are 
needed in an image database. 

And subjective keywords are to be distinguished from objective ones. Here is an example. A retriever wants 
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an image of “sea atmosphere”. In a database, one image represents the impression of sea, that is, “beach”. 
The other image represents the sea itself. When he/she specifies a keyword “sea”, the database system 
searches an image with keyword “sea”. It hits two images: the sea itself and the beach. However, what 
he/she really wants is the sea image, not a beach one. So that a subjective keyword should be given a 
subjective marker in the database, to distinguish it from objective one. The retriever finds the image more 
easily and precisely by putting in the keywords with a subjective marker. Such is the criteria of keyword. 



4 Examination and Assessment of the criteria 

We examined above mentioned criteria to assess the effectiveness. 

We arranged seven persons (5 males and 2 females; all adults). First of all, we showed them our criteria 
discussed in Section 3 as a book-style. We call it “keyword book’. Next, we show them 20 images, one per 
one paper. The contents of them are sight, animals, texture and so on. We included pictures of various image 
types. We scanned, digitized and printed pictures by a color laser printer. Then we let them give keywords to 
these images. They gave keywords to the images referring the keyword book as many as they liked. We did 
not limit the number of keywords for each image. 

Finally we let 3 specialists give keywords to same images separately. And we compared the former 
keywords (by 7 persons) with the latter keywords (by 3 specialists) so that we might assess the contents of 
both keywords. In the following section we discuss about the result of this examination. 

4.1 Number of keywords and the appropriateness 

The result of our experiment are presented in Table* • Total number of keywords is 416 for 20 images. In 
average there are about 21 keywords for an image. The relationship of keywords and object in each image is 
roughly as many. The specialists added total 1 03 keywords to those given 7 persons. 

It is the tendency that specialists give keywords to the small objects in each image. Especially there are a lot 
of objective keywords, which are given to images that has many objects. 7 persons and 3 specialists gave the 
same keywords to Figure* • But to Figure • *7 persons gave only 5 keywords: while 3 specialists did 29. 
And the keywords by 7 persons are different from the ones by 3 specialists. 

The keyword is for retrieval. We need not say that keywords should be given to as many as possible to 
improve the preciseness simply because the objects in the image are very small. Generally the more 
keywords the image is given, the easilier it can be searched out. But, on the contrary, from the point of 
appropriateness, we claim that the images should be added the information regarding the objects in it; for 
example, how large it is, or what impression a person has when he/she looks at it. This is to help the retriever 
to obtain the image which he/she really wants. The database should have such information. However, there 
is no information about them in our criteria. 

It follows that an index is needed to show the importance of keywords or square measure of objects in the 
image: for example, the keyword “monument” and importance “3”, or monument has one-fifth (of the 
image). 

To the images that cannot be dis tinguished by the object such as texture and patterns, the keywords of 7 
persons vary widely compared with ones by specialists. For example, in Figure • • 7 persons gave 
“red”, ’’water-drop” and “discomfort”. The object in it is so hard to define that they are difficult to give 
keywords to this texture and pattern. In educational situation, this is a serious matter because students are not 
generally able to hit such keywords to retrieve them. In short, we point out that metadata type is of limited 
use when images are given in pattern or texture style. So that it seems reasonable to say that feature type 
retrieve is better for them than metadata type. 

4.2 subjective/objective keywords 

In Table* • subjective keywords are 81, 21 % of total keywords. And the specialists gave 7 subjective 
keywords, which is 10 % of 7 persons’ keywords. On the other hand, the specialists gave 96 objective 
keywords. 
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Regarding subjective keywords, there is little difference between 3 specialists’ and 7 persons’. On the 
contrary, regarding objective keywords, 3 specialists’ are different from 7 persons’ . In addition to this, 7 
persons’ keywords are lacking of uniformity. There are only a small number of common subjective keyword 
that both 3 specialists and 7 persons. 

Therefore it follows that there are limited words that express the character of images subjectively. The 
number of subjective keywords using these limited words is so small. On the other hand, regarding the 
objective keywords, 7 persons mostly attached them to even the objects which occupy the main part of the 
image, while 3 specialists gave them to even the small objects. We may say that that caused the increase of 
their keywords. 





Total 


Subjective 


Objective 


Seven Persons 


313 


74 


239 


Three Specialists 


103 


7 


96 


Total 


416 


81 


335 



Table* -Number of Keywords 



4 3 Discussion about our criteria 

As mentioned above, we must modify our criteria because they will be to include the importance of each 
keyword. 

• Appropriateness* The database administrator should expect how the retriever uses keywords when he/she 
attaches keywords to the image. As mentioned above, he/she distinguishes or identify “car” with “BMW”. If 
the database is for general use, both general noun “car” and proper noun “BMW” are also given. 

• Number of keywords* K needs the number of object that the image has and at least one subjective keyword 
which express the characteristics of image. 

• Importance* It newly is appended our criteria. Importance: 3 is that the object is very large in the image or 
that it express the characteristics or the main theme of the image. Importance 1 is that the object is as small 
in the image as people try to find. The other object or character in the image is importance 2. For example, 
in Figure • • keyword: keyboard, importance: 1; keyword: PC importance: 2 and in Figure* -keyword: cat, 

importance^. 

• Subjectiveness /objectiveness* objective keywords should be given for all object in the image and subjective 
ones are given to at least one word for the impression by the image. Each keyword is appended 
subjective/objective marker or one s coded to numerically. 



5 Conclusion 

In this paper, we considered and the criteria of metadata, especially keywords for image database and image 
retrieval. We emphasized importance of consistency and appropriateness. We examined the criteria by 
experimenting 10 testers and verified them. And we added a new measures, “Importance” to our criteria 

When a lot of persons give keywords to images, such criteria are indispensable. In our future study, we will 
concerned and define the structure or the framework of metadata. Furthermore, We will integrate metadata 
type and feature type for pattern or texture. 

In the future, we hope that multimedia retrieval with metadata will be much easier for all people. So the 
teachers will be able to utilize more accurate images in education, and the students will be able to easily 
retrieve images from Internet. 
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Figure • • image with one object 
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Figure* • image with many objects 



Figure* • Texture 








The Research on Difficulty of 
Asynchronous Learning Materials Based 
on Studying Time Distribution 



Wu-Yuin Hwang and Rueng-Lueng Shiu 

Information and Computer Education 
National Kouhsiung Normal Univeristy 
wyhwang@nknucc. nknu.edu. tw 

The purpose of asynchronous distance learning systems is to enhance students' 
learning performance in the internet. In this paper, we investigate the 
characteristics of the asynchronous materials and propose the criteria to evaluate 
them. Employing the criteria, the materials could be adjusted to meet most 
students' learning pace. First, the TDC (time -distributed curve ) which is a 
learning curve is derived from students' studying time distribution. By the TDC, 
it is obtained that the more difficult the materials of the chapter are the steeper 
the TDC becomes. Also the total learning time of each chapter indicates the 
quantity of the matter. Employing the total time of each chapter, we could 
evaluate whether the quantity of the matter is sufficient to match students' 
learning desire. 

Keywords: distance learning* learning portfolio learning behavior bearing 
time distribution 



1 Introcduction 

1.1 The distribution of learning time with learning attitude 

Teachers could interact with their students immediately at the classroom. Thus, they could get the learning 
behavior of their students by students’ response. The learning behavior is regarded as a good measure to 
evaluate learning performance. But it is really hard to obtain every studenf s learning process and attitude 
because there are at least 30 students in each class. However, employing the database technology in 
asynchronous learning systems, it is possible to obtain all of the student’ s learning process and studying 
time. 

1.2 Learning time distribution 

In traditional education, students learning together in the classroom at the fixed time, and teachers control 
the course proceeding. But it is difficult to pay attention to all students. However, asynchronous learning 
systems not only provide a brand-new perspective to long-life learning but also keep track of learning time 
of all students. In accordance with the learning time of all students, teachers could modify the matter to 
match learning goals. 



2 Experiment and analysis 

The experimental course in our asynchronous learning system is “Basic computer concept”, the materials of 
the course are divided into 12 chapters. The progress -control mechanism is that students need to finish the 
homework of the chapter in order to be promoted to the next chapter. Thirty participants engage in this 
experiment and they are all teachers. 




The student* s learning time and login time are recorded by technologies of ASP(Active Server Page) and 
Database. Therefore, we could get which chapter students read and how long they read the chapter. The 
important curve, Time-Distributed Curve (TDC), is generated by linear regression analysis. From the slope 
and the area of TDC, some characteristics and results are obtained. 

2.1 TDC and DCA (Degree of Course Acceptance) 

Student* s reading time each chapter is recorded in our experiment. The recorded time begins from the date 
when the teaching materials are put in the internet for 15 days. In each chapter, all of the student* s learning 
time everyday is summed up. 

Employing the recorded data and derived chart, each chapter has a unique TDC (time -distributed curve) by 
linear regression analysis. According to the time -distributed curve, teachers may decide whether the 
materials should be improved. 




Fig. 1 The TDC of ch3 

Fig.2 Comparisons of the TDC of chapter 3,4 and 5. 



In Fig. 1, the X axle indicates time value and its time unit is one minute not an hour and The Y axle indicates 
days. For example, the total time on the 4 th day is approximate 150 minutes. The slope of the TDC is minus 
because the total studying time would decrease while students proceed to study the matter. 

The value of the slope is required to be concerned. The larger the value of the slope is, the smoother the 
TDC becomes. For example, figure 2 made comparisons of the TDC of chapter 3,4 and 5. Obviously, the 
TDC of chapter 4 has the smallest slope because it is the steepest one. And the TDC slope of chapter 3 is 
slightly larger than that of chapter 5. Thus, it is the most difficult to read chapter 4 and it is the easiest to 
read chapter 3. The reasons why the materials are hard to study may be either the materials are complicated 
or the user interface is not friendly to read. According to the above description, the slope of TDC could be 
termed as Degree of Course Acceptance (DCA, It means the harder the topic to read the smaller is the 
DCA.)* Besides the TDC* s slope is proposed to determine the degree of materials acceptance, there is 
another important characteristic, the area of the TDC, to influence the amount of learning time. 

Based on the area and slope of TDC, the difficulty and quantity of the materials could be evaluated. 
According to the above description, it is shown that the quantity of materials would affect the amount of 
learner* s studying time, also the difficulty of materials would affect the length of learning period. Due to 
these reasons, there are two margin lines, quantity and difficulty, in Fig. 3. The two margin lines are termed 
e are “Margin Line Of Quantity (MLOQ)** and “Margin Line Of Difficulty (MLOD)**. There are plentiful 
materials on the right of MLOQ, but there are poor on the left side. The upper of MLOD the materials are 
located the harder they are read, but lower are easy. 

Since the features of MLOQ, MLOD, DCA and the area of TDC are proposed, there are four kinds of 
situations that the TDC represents as follow: 

1. It is easy to read the material, and the contents are plentiful. 

2. It is easy to read the material, but the contents are poor. 

3. It is hard to read the material, but the contents are poor. 

4. It is hard to read the material, and the contents are plentiful. 
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The MLOQ and MLOD could be employed to enhance discriminating the difficulty of the materials if the 
DCA and the TDC's area of the chapters are different. Finally, how is the value of the MLOQ and MLOD 
obtained? The MLOQ is the average of all students’ learning time of one chapter. The MLOD is the average 
of all students’ learning days of one chapter. 




Fig.4 Compares the TDCs of the ch5 & 7 

2.2 Time distribution of the interdependent course 



What else may affect one’ s effort in the course? There are relationships between two topics. For example, 
there are relationships of dependency between chapter 5(Intemet I) and 7(Intemet II). Generally, the topic 
“Internet I” is dedicated to construct the fundamental concept and “Internet II’’ introduces the advanced 
ideas and practice. According to the normal teaching policy in both topics, the “Internet I’’ should have 
fewer and simpler materials than the “Internet II”. Thus learners spent much less time to study “Internet I” 
than “Internet II”, 

Fig, 4 compares the TDCs of the two chapters. As shown in Fig. 4, it is easy to find out chapter 7 has a 
smaller DCA (the slope of TDC), that is, chapter 7 is harder than chapter 5. Furthermore, the area of chapter 
7 is less than that of chapter 5, The TDC of chapter 5 is located at approximately 1 1 on Y axle and 600 on X 
axle and the TDC of chapter 7 located at 12 on Y axle and 280 on X axle. According to MLOQ and MLOD 
as shown in fig, 3, we concluded that “The chapter 7 is more difficult than chapter 5, but its quantities are 
much less”. It is different from we described before, “Internet I” should have fewer matters than “Internet II”. 
In our experiment, we provided much more contents in chapter 5 than chapter 7. Therefore the amount of 
materials in chapter 5 should be reduced. 



3 Conclusions 

The asynchronous learning service is an on-line collection of hypertext that provides us a new way to learn. 
Their students with different native intelligence come from any place and go to learn when they would like. 
It is very important to design and evaluate the asynchronous teaching matters so as to match teaching goals. 
This paper proposed some basic criteria to investigate the characteristics of teaching matters, then gave an 
advise to modify them to meet the learning desire. The basic criteria, the area and slope of TDC, are derived 
from learning time distribution. Through the basic criteria, instructors could modify the materials in 
accordance with most students’ learning pace and talent. Especially, our proposed mechanism is worth much 
attention to develop the adaptive learning system. Once the asynchronous learner’ s studying portfolio is 
available, the materials could be real-time adjusted to match the learner’ s state. 
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This paper gives short overview of a recently launched EU project for universal 
exchange of university course units for the higher education based on a 
brokerage and delivery system model. The basic components and characteristics 
of the approache are described. More attention is given to the approach that will 
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educational achievements. 
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1 Introduction 

The new technologies are transforming the ways business operate and the ways people work. They are also 
reshaping the expectations, needs, and opportuniies in education and learning. The customers of the 
education market are forcing the education to become demand-led, rather than production driven. The 
current technologies are providing basis for a new just in time, on demand approach the electronic 
educational products to be offered by virtual or classical universities through special platforms acting as 
intermediaries between the consumers and the suppliers of educational material. It is clear that the new 
tecnology alone will not make this new model of more efficient education to happen by itself. Rather, new 
innovative models of production, brokerage, delivery and presentation are needed that will put together the 
participants of the education process to collaborate globally and to use the advantages of the powerful 
technology. 

The European project »UNIVERSALrbrokerage and delivery system for exchange of university course units 
for the Pan European Higher education« 1 is one of the attempts this model to start to work. The project is 
part of the Vth Framework program of User friendly information society 2 funded by the Commission of 
European union. The project started in March this year and will last 3 years. The conzortium of the project is 
large as it encompassess 17 partners from EU 3 , among them twelve higher education insti tuitions from all 
over Europe, two research organisations, one SME 4 and two telecom service providers. The consortium has 
also partners from outside Europe as the model developed is expected to have global applications. These are: 
the Moscow research institut, HEC from Montreal andNanyang Technological University from Singapore. 

The model and the implementation in UNIVERSAL is based on an education brokerage and delivery system 
being developed to incorporate training by provision of valued-added services to both the customers of the 
system and the suppliers of the educational material. The major characteristic of the system is offer of 
different types of learning and delivery of the educational material and its distributive nature. There is no 
central site for delivery of educational material. Additional characteristic of the model is the system that will 
be developed for pan-european acreditation of the purchased and performed university course units by the 
participants belonging to institutions of the European higher education system. At this early stage of the 



1 UNIVERSAL -1ST- 1999- 11 747 

2 URL: http ://www.ispo.cecJnt 

3 EU stands for European Union 

4 SME- Small or Medium Size Enterprise 
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project is not possible to predict all expected circumstances and consequences that such system may have in 
the furure development of the educationa nad training. Brokergae approach in provision of distance 
education must first address a number of technological and educational ssues which are part of the 
UNIVERSAL work program. In depth discussion of the pedagogical paradigms that may take place as well 
as te number of technological issues of the work program are not possible to be discussed in this article 
because of space restriction. Here, we concentrate on the basic characteristic of the model; highlithing the 
implementation scenario and giving more information about the usability aspects of the technology that will 
be evaluated through monitoring of the educational. 



2 The model and the architecture of the system 

The model being developed is based on creation of an open, cross-border, educational market environment 
coupling brokerage and delivery of “live” and “packaged” 5 courses. The proposed framework 
accommodates and adds value to the various business models and course structures employed in European 
HEI 6 institutions. It will enable: 

• a single faculty wishing to experiment with the simple import of external material to enrich a specific 
course, 

• an existing alliance between institutions to make their exchange more efficient and to enrich it with 
types of course units not previously exchanged, 

• an Open University to extend the range and depth of its courses. 

The UNIVERSAL brokerage platform is an interactive hypermedia environment offered to the academics 
and administrators of European educational institutions to plan and select courses. It de-couple offers and 
course units provision on the supply side from enquiry, booking and delivery on the demand side. The most 
important element that enable this de -coupling is the catalogue of offered educational material and the 
supporting processes, that adequately describes all the properties, educational and technological of the 
course units. This approach is implemented as brokerage platform and a number of delivery platforms, see 
Fig.l. 




Fig. 1 The general architecture 

The brokerage platform is further divided in a customer part, a provider part and an administration part. In 
the customer part, a knowledge dialogue engine is responsible for the dialogue to the demanding institution 
or to the enquiring student. When interacting with customers, the dialogue engine establishes their 
background knowledge and guides them in the selection of a course unit. Prospective customers are 



5 Packaged courses are multi-media textbooks and WWW based courses, live courses are CWCS 
broadcasted courses 

6 HEI stands for Higher Education Institution 
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presented with choices according to a) pre-requisites and conditions attached to different types of courses 
(course profile engine), b) the suitability of different Institutions offering courses and c) the different 
delivery modes available for a particular course (delivery profile engine). Students will be able to make 
inquiries and express interest in course units that their home institutions have pre-selected and are willing to 
recognise and give credit. 7 In the provider part a provider dialogue engine is mainly used for feedback from 
the system back to the content provider. It is also used to handle the offer of content provision to the 
brokerage system to help academic and administrator users to plan new courses, submit course units, obtain 
customer records as well as learning progress profiles and assessment results. Demands for course units are 
sent to the administration engine, which looks for, offered course units fitting into the demand profile. The 
contract engine sets up the contract between offering and demanding side, thus it is dealing with registration, 
authentication and billing. In the administration part the system is driven by the administration engine, 
which is mainly a service database combined with a search engine. The tasks of observing and managing 
course scheduling and delivery issues, as well as timed interactions between the system and customers to 
enable the provision of joint courses are the responsibility of the delivery manager. The federation engine is 
dealing with distribution issues of the engineering implementation, like consistency, caching, forwarding of 
requests. 

For each class of course unit there is an appropriate delivery platform. Variations in delivery platforms are 
due to the nature of the interactions and differences h media content and formats. The project will 
implement a limited range of delivery platforms, linked to the brokerage platform, sufficient to show the 
potential of the model. Each delivery platform contains an inherent delivery model and specific elements 
dealing with the media content itself 



3 The technology applied 

The technology used is ubiquitous, Internet based, offering common, portable solutions and large-scale, 
shared, cross-border systems. These systems include, but are not restricted to: 

Brokerage system is WEB based with Internet links. It is a central point of information, control and 
administration and logically will be centralized which means that only certain parts will be distributed or 
redundant for reasons of performance and high availability. The brokerage system will essentially be an E 
business system that uses technologies like XML, Java / RMI, CORBA / ODP traders or agent based 
systems, standard security technology and intrinsic service negotiation for content delivery. Advanced 
transaction and billing functionality based on AAA 8 security technology, implement sophisticated 
administration and monitoring interfaces to the delivery systems and the integration of assessment 
capabilities 

Delivery system similarly to the brokerage platform consists of an existing product/technology plus some 
enhancements (“glue” or “shell” around it), that acts as mediator between the delivery system and the 
brokerage platform. Defining a unique architecture for this interfacing allows adding additional systems / 
products by just implementing the appropriate interfaces that plug into the general architecture. A delivery 
system within the general architecture (see Fig.l) consists of content source, network and content sink. The 
content source can be a standalone system or part of a broader platform. The same applies to the content sink. 
The following delivery systems previously tested and used will cover the required functionality: Non-real- 
time delivery systems: retrieval of non-reaktime mono-media and multimedia contents (e.g. browsing 
through text and hypertext information, download of content files). Real-time A/V systems: Real-time 
retrieval of multimedia contents (stored and live contents including A/V material like MPEG-2 streams) in 
synchronous and asynchronous manner (on-demand, live and scheduled broadcast scenarios) using IP 
technology (unicast, multicast), and broadband technology (ATM, ADSL,). CSCW systems: 
Videoconferencing and Computer supported co-operative work (CSCW) to enable interactive forms of tele- 
lectures combining parallel transmission of A/V streams and course material (e.g. slides) with the possibility 
to interact with the lecturer as well as with other parts of the audience. This family of delivery systems shall 
also support real-time experiments, simulations and case studies. 



7 Scheduling of actual course units remains the prerogative of the institutions and booking of places on 
courses is considered to be an internal function of the institutions 

8 AAA stands for Authentication, Authorization and Access Control 
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All delivery systems are inter-working with the brokerage platform and content provider systems to 
synchronise announcement and content delivery, to guarantee controlled user access, and to manage 
selection, compatibility and resource usage in delivery. All supported delivery modes will be available as 
profiles of the A/V delivery family. This means that each content provider can easily select the appropriate 
profile according to the nature of his contents and his network resources. It is important to note that used the 
A/V delivery technology is not based on the current average bandwidth and quality availability of the 
ubiquitous Internet. It will offer real high quality of A/V contents as broadband communication over the 
Internet (with technologies like ADSL) for a relevant number of users is available in Europe as well as the 
broadband services emerging in the convergence technology market (broadband over Satellite or cable -TV, 
interactive digital TV services) are expected to boost the widespread usage of broadband A/V information in 
the global IT environment. CSCW technology applied follows the principle of standards compliance and 
openness as for example the powerful CSCW tool ISABEL, developed in the RACE and ACTS projects 
ISABEL and NICE, then the standards-based (e.g. ITU H.323) COTS products (e.g. MS NetMeeting), 
MBONE multimedia conferencing tools (VIC, VAT, WB) developed within project MECCANO etc. 



4 Content provision and description 

The general architecture and model envisaged as a point of interaction of “sellers and buyers” on one hand, 
and of “place of commerce for actual content” requires an intelligent abstract description of the contents. 
Therefore, meta-data standards for multimedia contents and for educational environments are applied in the 
catalogue building of course units. The meta data system used is based on the specification of the IEEE 
LOM 3.8 meta-data scheme 9 with some extensions relevant to the platform developed as for example: 
attributes that specify the copyright and IPR 10 protection, attributes that describe information about digital 
signatures, watermarking, attributes that describe the network requirements for provision of appropriate 
quality of service, attributes relevant to live content and attributes defining the type of the course unit which 
may be: packaged, live, CSWC or mixed. The content provision and course description is based on a meta- 
dada system selected from available standard documents or previous projects results (1). 

Several projects, that have investigated the management of information retrieval and the utilization of 
metadata for education and training have already proposed sets of meta-data requirements, like the 
Instructional Management System (IMS) project in the US (2) or the GESTALT (3) project in Europe. Some 
of the proposed sets have been evaluated and selected for the use in the UNIVERSAL project. 

They are: 

• Learning resource content meta-data: 

Learning resource content meta data that enables cataloguing of contents of arbitrary aggregation level. 
UNIVERSAL supports the following granularity levels: Fragment (Course Unit), Lesson , Module and 
Course . Each learning resource submitted by the “seller” is classified according to the aggregation levels 
and is added to the UNIVERSAL catalogue. UNIVERSAL supports "packaged" learning content, which is 
asynchronous in nature and synchronous learning content. Synchronous learning content is delivered as live 
transmissions of lectures, optionally supplemented by synchronous group ware communication technology. 
The special or unique features of the live content is described by specially developed attributes that are not 
part of the current existing meta-data standards. 

• Course structure meta-data: 

The UNIVERSAL brokerage platform not enables “buyers” to locate, use and re-use single course units. A 
functionality of the brokerage platform enables combination of single course units into higher levels of 
aggregation e.g. for full subject. This allows production of “custom” tailored complete courses. This is 
reflected in the course structure meta enabled by Course Structure Format (CSF) defined by the AICC and 
the ADL (1). 



9 IEEE Learning Objects Metadata document by WG 12 of IEEE Learning Technology Standardization 
Committee, URL:http://ltse. ieee.org/wgl2/scheme.html 

10 IPR stands for Intellectual Property Rights 
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• Contents packaging meta-data: 

Like many other LMS, the UNIVERSAL platform offers the possibility to access packaged course units, 
which are mainly pre-recorded. Packaged courses enable an interaction with the contents itself, the 
interaction with the lecturer is usually not offered. UNIVERSAL packaged courses are described according 
to well-defined rules, specified e.g. in available standards like the IMS CPS (Content Packaging 
Specification). 

• Descriptive information about assessment procedures: 

Assessment is an important concept in on-line education and learning. Although the UNIVERSAL platform 
is not designed to assess the student’ s advancement and the learning achievement, it will enable consumers 
or “buyers” to assess the functionality and usability of the platform and the delivery process. The assessment 
is performed with on line involvement of the students. Student’ s responses will be captured using some form 
of structured mechanism with designed template(s) for such purpose. This is especially required in the cases 
when questions in the templates involve multiple choice, matched items, fext selection, etc. Several 
structures exits for this purpose already: QML, SATML, and the IMS "Question and Test Interoperability 
Information Model" (QTI), UNIVERSAL team will decide which of the proposed template will be used for 
particular part of the platform and the delivery system used. 

• Meta-data for synchronisation: 

The asset management is important if the system is built up from re-usable learning resources e.g. units of 
lower aggregation level as is the case with UNIVERSAL catalogue. . In such cases the lifecycle 
management of the unit must be supported e.g. a component is not deleted from the asset management 
system in cases when the course unit is added to a module or larger course. This property of the system is 
described within the meta-data for synchronisation. 

The UNIVERSAL consortium brings together a selected group of engineering and business schools across 
Europe who are interested in pooling their academic resources for the purpose of broadening the choice of 
courses and pedagogical materials offered to students and teachers and to provide opportunities for 
international collaborative learning. Although a few of its members have already established bi-lateral 
academic exchange programmes, in the initial phase of the project, partners are collecting course units 
developed within particular partner or outside in order to build a catalogue for the greatest market potential 
for the brokerage platform. The catalogue contains in addition to the meta-data information also: brief 
description of the educational objectives of the academic content for the students to whom courses are 
offered. In addition to that, brief description of teaching methodology used at each institution, description of 
the academic calendar for each institution, and description of the academic accreditation process for each 
institution are also information provided in the catalogue. Currently course units are collected from the 
following fields: Introduction to Information Systems, Regional Economic Development and 

Telecommunication, Global Marketing Management, Business Case studies, Metallurgy simulations and 
experiments, Foreign Languages and Cross-cultural Behaviour, Statistics for Economist, Tele -management. 

The UNIVERSAL brokerage platform enables to locate, use and re-use single course units and combine 
single course units into higher levels of aggregation. This approach makes possible a definition of custom 
tailored structure for a complete course, which will lead, to a definition of a standard system of granularity 
for learning resources with the other projects from the 1ST cluster “Flexible University”. 

Several institutions already made suggestions for a hierarchy of aggregation levels. The IEEE LTSC and the 
IMS so far define four levels of granularity: Fragment , Lesson , Module and Course . Some other US based 
systems of aggregation can be found in the bibliography enclosed (8). 

A course structure representation defines all of the course elements, the course structure, and all external 
references necessary to represent a course and its intended behaviour. The ADL together with the AICC, 
IEEE and IMS have developed the so-called Course Structure Format (CSF) which was the recommended 
approach for the UNIVERSAL project. The CSF promotes reuse of entire courses and encourages the reuse 
of course components by exposing all the details of each course element. The CSF is intended to reduce or 
eliminate dependency of a course on a particular LMS implementation. 

The CSF is also intended to represent a wide variety of course structures and content "aggregations". 
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Content structures can be represented by the CSF that range from very small "chunks” of content - as simple 
as a few lines of Hypertext Markup Language (HTML) or short media clip - to highly interactive learning 
content that is tracked by an LMS. The CSF is neutral about the complexity of content, the number of 
hierarchical levels of a particular course (i.e., "granularity"), and the instructional methodology employed to 
design a course. 

The UNIVERSAL platform incorporates continuous assessment of content and the system itself based on 
the interactions between the customers using the delivery of course units and the system itself. This 
information will be used to improve the overall scheme and the content to teachers, administrators and other 
possible customers. 



5 The usability and evaluation 

“Evaluation is the activity that throughout the planning and delivery of innovative programs enables those 
involved to learn and make judgement about the outcomes of the innovation concerned “ . The 

UNIVERSAL project aims also to develop tools to monitor the innovation process of education and learning 
and to develop best practice guidance. 

The assessment of the content and the overall system, components for functional assessment are 
incorporated in both the brokerage system and in each of the delivery systems. Results from previous 
projects including the deployment of trans-national multimedia learning schemes 12 have shown that it is vital 
that all participants involved in the creation of the exchange platform and its educational content have a 
mutual understanding of the platform* s operations, functions and of the components* interaction. To fulfill 
this goal it is necessary to give the users the tools enabling an easy the use of the exchange platform such as: 
an administrator guide, oriented towards the management of the platform, from a technical (“how to use”) 
point of view as well as from a content (“what to do”) point of view; a user guide, describing the day-to-day 
use of the system e.g. to a teacher wishing to use content available through the platform (how to access the 
catalogue, how to book a live course, etc). In addition, in the case of “live” delivery systems (on-line live 
courses with CSWC), experience have shown that it is necessary to organise “hands-on” training sessions to 
free the teacher from the fear of new tools and to strengthen their “moderation” skills when working with a 
geographically distributed class through a TV-like systems. Classes in the live courses will be mainly cross- 
border meaning that most participants will not be working in their mother tongue and there will be a mix of 
cultures present in any one of the classes. As part of the preparation for participation in the main trials a 
short “Language & Behaviour” courses that will (I) help participants with their colloquial English (since the 
majority of the courses will be held in English) and (II) help them be aware of differences in cultural 
behaviour, e.g. questioning style will be provided. 

The student/teacher ratio varies greatly among the UNIVERSAL consortium partners. A set of software 
monitoring tools are being studied to be implemented into the brokerage platform to make it possible for 
students to continuously assess their personal progress and to choose the academic path best adapted to his 
or her acquired knowledge and skills; for course unit providers tools will be used to improve the 
effectiveness of programmes offered to learners and modify content accordingly; tools will be used also to 
analyze the way learners use the courseware provided; to enable a global assessment of the usability of the 
platform etc. The monitoring tools as well as the usability evaluation techniques used for assessment of the 
innovation technology approach are being developed/selected in accordance with the ACTS Usability 
Evaluation Guidelines 13 . These guidelines define the testing and evaluation methods, experiments design, 
definitions of interviews, observations, heuristic evaluation and surveys. 

The evaluation instruments for courseware evaluation and corresponding measures will include: 

• Pre -task/post-task questionnaires 

• Task experience questionnaires 

• Computer experience questionnaires 

• Exams or assessment of performance 

• Post course questionnaires 



1 1 Stem(1988) in Evaluation of Learning Technology Implementation, by Barry Jackson, 
URL: http://ltdi.icbl.hw.ac.uk 

12 Project LEVERAGE AC 109 from the IV Framework Program- ACTS 

13 USINACTS - AC 224, IV Framework Program -ACTS 
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• Knowledge quizzes 

• Logs - logging times 

The approach for educational assessment will follow the practical guidelines developed within the LTDI 
technology developed within the Learning Technology Dissemination Initiative funded by the Scottish 
Higher Education Funding Council 14 . As a result of this a set of guidelines identifying best practices for 
future users of an academic brokerage platform will be produced. Academic partners will have the 
possibility to review the existing experiences and pool their lessons learned from prior involvement in 
flexible, distance, and collaborative teaching and learning programmes. This information cover issues such 
as the choice and format for resources provided to students and teachers, access to tutors or teachers, 
methodology, independent study and collaborate work, learner motivation, learner monitoring, course 
accreditation etc. 



6 Conclusion 

The UNIVERSAL is a project that implements the EU policies regarding the development of the European 
higher education and the user-friendly information society in particular: 

• By improving the quality and diversity of the pan-European HE system 

• By promoting the globalisation of the exchange of HE course units 

• By enabling partners from economically disadvantaged regions, particularly in Central and Eastern 
Europe to participate in these developments and helping them to strengthen and enrich their course 
offerings and the foster the education in general. 

Most of the activity within the project will be tightly connected with the usability aspects of the applied 
methods and technology. Usability evaluation and proposed improvement will be based on the past 
experiences, guidelines and standards developed within projects that have addressed this issue of modem 
technology in depth. The consortium expects wide acceptance among the higher-level education institutions 
in Europe. 
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Computers have been wildly used in elementary schools and to input characters 
is usually necessary while manipulating computers. However, inputting Chinese 
characters is a burden for Taiwanese pupils. Keyboarding is a traditional method 
used for inputting Chinese characters. On the other hand, other user- friendlier 
input tools like speech and handwriting recognition provide alternative choices. 

We observe three 12-year-old pupils, who have different backgrounds, how they 
use these different methods to input Chinese characters. Experiments results 
show that all three pupils make progress after short-time practice for all input 
methods. Different pupils, however, would choose different input method 
depending on their own preference and background. 

Keywords: Chinese input method, keyboarding input by pronunciation, 
speech recognition, handwriting recognition 



1 Introduction 

Computers have been widely used in many educational applications like computer-aided learning. For a 
pupil in Taiwan, one of obstacles that he or she may encounter for using computers is to input Chinese 
characters.[l] In this study, we intend to explore pupils’ behaviors on Chinese characters inputting and then 
make recommendations in under what conditions what kind of input methods may be u sed. 

Keyboarding and non-keyboarding are two main categories of characters input methods. For keyboarding 
method, users type Chinese characters either according to their pronunciations or by "dismantling" the 
characters. For the non-keyboarding methods, handwriting and speech reorganization are two popular 
methods for inputting Chinese. While keyboarding has been used for longer time and almost always each 
computer is equipped with a keyboard, non-keyboarding input methods are developed recently and extra 
equipments and software are needed. 

When keyboarding is used, most students input each single Chinese character by spelling its 
pronunciation.^] Main reasons are: (1) Students are familiar with the pronunciation. What they have to 
learn is to memorize the position of each key on the keyboard corresponding to the pronunciations. (2) 
Psychology indicates that human beings think in the form of tone (of characters), but not the font of the 
characters. [4] However, disadvantages of this input method exist. (1) Students may not spell pronunciation 
correctly especially when they encounter new words. (2) Many different Chinese characters have same 
pronunciations. [3 ,7] Pupils thus need to choose the target character and the input speed is slowed down. 
Non-keyboarding input methods, especial the voice reorganization, is one of the hottest research topic in 
computing. People use natural ways like handwriting or speech to communicate with computers. The 
computers then analyzed the data to identify what people mean and output the data in a text format.[6] 

We conduct the study to find out pros and cons of each input methods. Three sixth grade (12-year-old) 
students with different computing backgrounds are chosen, based on interviewing and simple tests. Student 
A has a lot of computing experience, student C has little computing experience, and Student B is in between. 
All three students come from National Tainan Teachers College Affiliated Primary School. For keyboarding 
input, they use the input method of spelling words' pronunciation that is included in the Windows 98 
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operation system. For handwriting reorganization, they use "Pen Power Jr." For speech reorganization, they 
use the product named "IBM ViaVoice." The essay that students input is extracted from the newspaper China 
Times. There are 142 characters in the essay. To evaluate the performance of different input methods, each 
pupil has five chances to input the Chinese essay by pronunciation based keyboarding, speech reorganization 
and handwriting reorganization. Between each of chance, they have a period of 30 minutes to practice. In the 
next section, we summarize our observations and statistical results toward the problems in each of the sub- 
sections. 



2 Findings and discussion 

2.1 Can short-time trainings improve performance of pronunciation based 
keyboarding? 

Figure 1 shows timing in seconds that student A, B, C took to finish the essay in five trials. They are asked 
to input all characters correctly. All three pupils made progress for the short-time practice. Comparing the 
first and the fifth performances, we found that student A, B, and C had 19%, 37%, and 53% of progress, 
respectively. Although student A made less progress, the student actually finished the essay quickest. Beside, 
students A, B, and C had an average progress of 16, 60, 178 seconds. We noticed that both student B and C 
kept making progress, but student A had a downgrading performance from the third to the forth trial. The 
difference between the third and forth trial for student A, however, is not significant. Based on the 
experimental results, we conclude that short time training does benefit three students in different background. 
Furthermore, the student with the least computing background actually made the most significant progress. 

Figure 1. Comparison of input timing using pronunciation based on keyboarding. 



□ Student C 
EJ Student B 
■ Student A 



• ••• time* 




2.2 How does the speech recognition input method affect the input efficiency? 



Figure 2 shows the percentage of the speech recognition software correctly identify the characters. Both 
student A and B perform similar from the first to the fifth trial, while student B tends to be better in the first 
of four trials. Besides, student B makes a great progress from the first trial (77%) to the second trial (92%). 
Student A and B reached a 100% correct recognition and student C achieved almost 90% correctness. This 
suggests that, after reasonable training, students with different background can manipulate the speech input 
method nicely. 

Figure 2. The correct recognition percentage of speech recognition. 
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■ Stuednt A 
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Figure 3 demonstrate timing the pupils took in five trials with emending the wrong characters. When the 
correct recognition rate is high, the input method can be very efficient, about 160 characters in one minute 
for student A and B. However, this is not a universal situation. Take student C as the example, the pupil 
speaks in such an ambiguous tone that the computer simply could not correctly recognize the essay. As the 
result, the student spent a lot of time to emend the wrong characters and greatly slow down the speed of 
inputting. Such observations show that there is room for improving the correct recognition rate. 

Figure 3. Timing of speech recognition input. 
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2.3 How does the handwriting recognition input method affect the input efficiency? 



Figure 4 reveals the percentage of the handwriting recognition software correctly identify the characters. 
Both student A and B perform similar from the first to the fifth trial, while student B tends to be better. 
Besides, student B reached 100% correct recognition at the last two trials, and student C achieved 92% 
correctness. This suggests that, after reasonable training, students with different background can manipulate 
the handwriting recognition input method nicely. 

Figure 4. The correct recognition percentage of handwritten recognition. 
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G Student B 
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Figure 5 demonstrate timing the pupils took in five trials with emending the wrong characters. We find that 
both student B and A perform similarly and nicely, though slight differences exist. Main reason why student 
B outperforms other two students at the correct recognition percentage is the way he wrote Chinese 
characters. Student B usually writes in a way that the characters strokes are clear and distinct. In contrast, 
student C tends to write Chinese characters apart and thus make the software consider as several words. 
Furthermore, student A writes very fast and his written characters strokes are also cleai; though a little 
behind the ones of student B This characteristic affect the timing of input, while the correct recognition 
percentage of student A is lower than B, he should took more time in inputting, instead, the timing of input 
with emending is less than that of student B. 

Figure 5. Timing of handwriting recognition input. 
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2.4 For different students, how the input methods help students to input more 
efficiently? 

For student A, to use keyboarding as input tool is better than the one by non-keyboarding. Because he is 
familiar with keyboarding and input Chinese even becomes a basic ability as using computer. [2] So that in 
the test of keyboarding, he exceeded others very much at first, but just for the same reason, he could not 
make further progress. In inputting by non-keyboarding methods, he did not do such a good job as 
keyboarding. Take speech for instance, everyone spent almost the same time when they did not need to 
emend the wrong words, but he spent much more time then student B when emending is needed. Similar 
result can be found for handwriting method. The recognition rate of student B is better than A. 

For student B, to use non-keyboarding as input tool is better than the one by keyboarding. One reason is that 
he is not as familiar with keyboarding as A does. He usually uses computer to play games or surf the Internet. 
He actually just needs to use mouse as a tool to communicate with computer. In keyboarding, although his 
effect is not better than student A, but he has a better progress than student A. If non-keyboarding method is 
used, no matter speech or handwriting, he did a better job than student A and C. We emphasize that, these 
three students never use speech or handwritten before, and the result shows that the learning effect of student 
B is better than the other two students. 

For student C, to use keyboarding seems better than non-keyboarding. He made a better progress in 
keyboarding. In input method of keyboarding, at the last time of test, his input time was even less than that 
of student B’s first trial. This result showed that after short-time training, student C made the greatest 
progress in keyboarding (53%). For non-keyboarding methods, student C seems had difficulty to adjust his 
speech tone and handwriting to fit the software. But we think this is the place that software developers 
should work on. It should be the software developers’ responsibility to develop software that is capable of 
adapting different speech tone and handwriting habit. 



3 Conclusions 

We have observed the behaviors of using keyboarding and non -keyboarding Chinese input methods. We 
suggest the following. For students who are familiar with computer and takes inputting as a basic skill, like 
student A, keyboarding best fits. For those who learn new things very fast, are interested in them, and speak 
or write well, (like student B) then the non-keyboarding is fit. For a student like the C, who does not learn 
new things very well and maybe he does not speak or write well, and he is used to input by keyboard, then 
the keyboarding is fit. 
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Xtrain is a program for scripting and presenting multimedia displays. This 
program was developed in the Advanced Learning Technologies Laboratory at 
the University of Memphis and has been used in variety of psychological 
experiments. This program can combine such multimedia formats as Microsoft 
agent, Macromedia flash, director and many others that are available for 
scripting under a GUI Windows environment. Furthermore, Xtrain offers a 
variety of options for testing styles. 
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1 Introduction 

The idea of embodied conversational agents has generated considerable interest in the realm of HCI recently. 
Unfortunately, for the most part this has been metaphorical, because computers could not support the needed 
software. In order for the computer to fully support embodied conversational agents, they would need 
software that could produce and control many human-like characteristics, such as conversational behaviors, 
with the ability to mediate the flow of conversation by the use of such things as facial expressions, hand 
movements and voice intonations [1]. 

Microsoft made one such attempt at this with their Microsoft Agent program. Microsoft Agent is an 
interactive interface with conversational capabilities that are embodied in an animated character agent. 
One example of this is the helper agent in newer versions of Microsoft products such as Microsoft Word and 
Microsoft PowerPoint. Microsoft Agent is also an optional program for windows 9x operating systems and 
is available for free download at the Microsoft site. It is compatible with all MS Windows platforms 
starting with Windows 95. Thus, this agent is readily available for widespread use [5]. The purpose off 
this paper is not to review Animated Agents for a review, see Johnson, Rickel, and Lester [4]. 

Furthermore, recent research has shown that the correct use of multimedia presentations can enhance the 
learning and memory from presented materials. Multimedia in this case refers to any type of pictorial 
information presented with textual information. However, this form of learning works best with pictorial 
information shown as an animation that is then coordinated with a narration of any textual information that 
would be needed [6]. Under the cognitive theory of Multimedia learning, there are three main rules that 
should be considered for scripting of a multimedia presentation: Spatial contiguity, Temporal contiguity, and 
Modality. The spatial contiguity effect states that relevant and related concepts should be presented in the 
same general area of each other (e.g. labeled words should be closer to the object they label than other 
objects on the screen). The temporal contiguity effect informs us that the various forms of media used 
during a presentation should correspond with each other by occurring at the same time. Lastly, the modality 
effect says that if two types of information are presented in the same type of modes, it will hinder learning. 
However, this can be overcome by presenting information in two modalities. So, printed text and 
animation on a computer screen would be a hindrance to learning, but a narration and an animation would 
not [7; 8]. 

Since Microsoft launched the first version of Microsoft Agent, users and developers have provided a lot of 
resources for use with the program (e.g., some information can be obtained 



fron bttp://www.msagentring.org/agentringmain,html) . There are several innovated approached both in the 
use and the scripting of Agents. For example, mash.exe fottp://www.bellcraft.com/mash/ ) provide a very 
useful scripting tool for agent programming. Many of these programs have been examined, including Mash, 
and while they have the ability to control Microsoft Agent, they are lacking the ability to synchronize the 
Agent program with other forms of multimedia. 

The Advanced Learning Technologies laboratory at the University of Memphis developed Xtrain as a way to 
incorporate embodied agents (Microsoft Agent) and other forms of multimedia into instruction, research, 
and presentations. Psychologists have used products such as Mel©and Super lab©to run experiments, but 
these programs cannot incorporate newer technology. Xtrain provides ways to script many different kinds 
of presentations, including Microsoft Agent, audio and video clips, HTML, Macromedia flash files, 
Macromedia Director files, and many graphics file formats [2, 3]. 

This software program serves a duel purpose. It is both an authoring tool and presentation tool. These 
work together to form a powerful and versatile tool for the presentation of various multimedia displays as 
well as data collection. 



2 Authoring tool 

The authoring side of the program has two levels (a) overall organization of frames and (b) detailed 
construction of individual frames. The overall methodology is similar to the SuperLab program used in 
experimental psychology. The Presentation is organized in terms of a tree structure with each node in the 
tree as a pointer to presentation frames. Each frame consists of the smallest unit of information and the 
frames are logically contingent upon each other. Such tree structure serves as basic navigation guidelines. 
However, the navigation path can be quite flexible depending on the needs of the user. The tree structure 
can be created using a user friendly GUI. Each frame corresponding to the tree nods can be any of several 
formats such as text art, pictures with hotspots, video/audio clips, agent interactions, and animations. 

Xtrain has extensive options for frame editing. The program has been arranged so that the different editing 
functions displayed as individual property tabs. Each tab corresponds to a specific multimedia format. A 
description of the property tabs will follow. 

Property Tabs 

2.1 Frame Property Tab 

The Frame property tab allows the basic outline of the frame to be determined. From here frame duration 
is set, alone with the frame’ s properties, and the frame type. The duration can be anywhere from self-paced 
to any amount of time desired measured in milliseconds. The type of multimedia desired can be selected 
under a Frame properties drop-down menu. Under the Frame type dropdown menu, the type of frame can 
be specified: Normal, Title, Review, Test, or Interaction. 

2.2 Agent Property Tab 

This is the general tab that is used to control the agent. Each frame can have up to three agent actions 
assigned to it. These actions are denoted as agent 1, agent2, and agent3. However, these can be assigned 
as needed for example one agent can be given as many as three actions or three agents can be given one 
action each. These are selected from the available agents using the Agent dropdown menu. Just below 
this dropdown menu is a dropdown menu that specifies when the agent will be used. For example, “Action 
over frame” can be selected so the agent is active while the rest of the frame is running. Just below this are 
three additional tabs that specify (a) the agent's position on the screen, (b) what the agent will say in each 
frame, and (c) balloon formatting, if the agent has this option. These are the Action and Gesture tab, Speak 
and Play tab, and Balloon Setup tab, respectively. 

Of these tabs, the Speak and Play tab is of the most importance. This frame in its most basic form allows 
for text to be entered into a text box. The agent reads this text using a text to speech engine. However, 
this text box can also contain simple markup within the text. This markup includes such speech parameters 
as volume, emphasis, pitch and speed. These markup tags can be inserted into the text by inputting the 



desired values into the box beside the parameter name on the right portion of the tab and then double 
clicking the name. This list of parameters also includes a few special tags that can control the flow of the 
information delivery. These tags permit the agent to skip to a specific frame in the tree structure (Show 
Frame), or to go to specific frames in a selected Shockwave Flash movie (Go to Frame in Flash Movie). 
The remaining tag option is Insert Special action. This set of tags allows the user to start, stop, and restart a 
flash movie, and provides a tag that terminates the program at the end of a presentation. The Speak and 
Play tab allows for assignment of actions to the selected agent. These actions vary according to the 
abilities of the selected agent, and can be assigned either at the beginning or the end of the text the agent 
speaks. 

Similar to other agent scripting tool, such as MASH, this agent property editor uses all available Microsoft 
agents controls. In addition, Xtrain utilizes the bookmark function of MS Agent to control the overall flow 
of the presentation. In fact it is the use of these bookmarking functions that make it possible to control 
Multimedia synchronization, such as with Flash animation, which is lacking in the other agent programs. 

23 Text Display Tab 

The text display tab is used to insert text to be displayed on the screen. Doing this involves clicking on the 
display area, typing in the text to be displayed, and then clicking update. The text will then appear in the 
display area in the same way that it will be displayed on the screen during the presentation. 

2.4 Multimedia Tab 

The Multimedia tab allows you to assign audio files, movie files, and wallpaper to the frame. The program 
supports wave files (.wav) and Enhanced Linguistic files audio formats. If an Enhanced Linguistic file is 
used Microsoft Agent can be made to appear to speak the file. The movie files available from this tab are 
AVI (.avi) and Mpeg (.mpg). A Bitmap (.bmp) image can be set as a background that either covers the whole 
screen or centered. 

2.5 Pictures Tab 

Using the picture tab, a picture can be added to the frame and manipulated. Xtrain supports two types of 
graphic files: Bitmap (.bmp) and GIF (.gif)- The picture can be located at any point on the screen, centered, 
or can move from point to point. A hotspot option can be added to the picture to be used to give commands 
to the agent or to play audio files. Each hotspot can have information, such as text and tagged markup, to 
be sent to any selected agent. 

2.6 Shockwave Tab 

Under this tab, there are two options: Flash Movie and Shock Wave Movie. Flash movies and shockwave 
animations are among the most frequently used multimedia format. Xtrain uses activeX control from 
macromedia so both types of movies can be manipulated. By loading flash movie from this tab, detailed 
frame information can be examined so Agents can navigate through the movie. In addition, Xtrain uses 
FSCommand of flash movie to control Agent and the tree navigation. 

2.7 Frame Summary Tab 

The frame summary tab gives summary information both at the scripting phase and at the presentation phase. 
At the scripting phase, it gives a brief overview of the selections made in the other tabs for that frame. If the 
frame is a test frame, it also contains the correct answers to the questions given in the test frame. After 
viewing on the other half of this frame, responses are shown. If it was a test frame, the student’ s responses 
are listed along with whether the response was correct. 

2.8 HTML Tab 

The program allows for the incorporation of html documents into presentations. This allows greater 
flexibility in terms of specialized displays. The format allows for html documents that are locally saved in 
the Xtrain directory to be displayed and navigated during presentations. 
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2.9 Test Tab 

One other important feature of Xtrain is the testing option. During the scripting phase, frames can be 
assigned as testing frames on the frame property tab. These frames can be used to capture information 
from the user. They allow input in such forms as multiple -choice questions, short answer questions, and 
even essays. At the end of the presentation phase, input from the participant is automatically saved as an 
ASCII text file. The agent can also be programmed to give dynamic feedback, when the participant gives 
wrong answers. 



3 Presentation Tool 

The presentation of the scripted material is as easy as selecting the run drop-down menu and selecting the 
run entire session option. Alternatively, the Xtrain presentation file (.xtr) can be ran by double clicking its 
icon in the strain folder. This action occludes all other objects on the screen: only the scripted presentation 
and a control bar are visible. This control bar is a flash file that allows for the following actions: go back, 
continue, help, and progress. The presentation continues forward until it reaches the end of the 
presentation. 



4 Summary 

Xtrain is a program that is able to integrate multimedia files into one presentation format. The authoring side 
of the program takes advantage of many Windows’ standards for ease of use. It provides a standard 
Windows interface window with icon buttons and drop-down menus, such as File, Edit, Window, and Help. 
These offer such options as open and save in the File menu, as well as, cut, copy, and paste in the Edit menu. 
Xtrain also offers a special drop down menu labeled Run. This menu offers the options of running the entire 
session or of previewing a selected frame. See Figure 1 for a view of the program. The frames are structured 
in a tree format that is located on the left of the screen. This tree is created via buttons labeled Brother, for 
frames on the same level, and Child, for frames on a branching level. Each frame can be scripted using nine 
different property tabs: Frame Property, Agent Property, Text Display, Multimedia, Picture, Shockwave, 
Frame Summary, HTML, and Test. These tabs may be individually associated with each frame. It is from 
these components that the script is produced to set the required tone for the information to be presented. 
Microsoft agent can also be used to control the flow between frames, so that if the need arises the agent can 
direct the presentation to any frame in the tree. Furthermore, if a Shockwave Flash file is used, the agent also 
has the ability to direct the flash movie to any frame within the movie. These options allow for maximum 
flexibility for the user when scripting a multimedia presentation. In addition to this freedom in scripting, 
Xtrain offers an easy presentation method that either selecting run entire session from the run menu or by 
simply double clicking on the created Xtrain file. 



References 

[1] Cassell, J., Bickmore, T., Billinghurst, M., Campbell, L., Chang, H., Vilhjalmsson, H., Yan, H., 
“Embodiment in Conversational Interfaces: Rea.” Proceedings CHI’ 99. Pittsburgh: ACM. (1999). 

[2] Craig, S. D., & Hu, X. Xtrain: Expert training and data collection system. Presented at the 1999 
meeting of The Society for Computers in Psychology. Los Angeles, California. (1999, November). 

[3] Hu, X. “Xtrain 1.0 [computer program].” The University of Memphis: Author. (1998). 

[4] Johnson, W. L., Rickel, J. W. & Lester, L. Animated pedagogical agents: Face-to-face interaction in 
interactive learning environments. International Journal of Artificial Intelligence in Education, Vol. 11, 
in press, (2000). 

[5] Microsoft. Microsoft Agent 2.0 [computer program]. (1998). 

[6] Mayer, R. E. Multimedia learning: Are we asking the right questions? Educational Psychologist, 32^ 
1-19,(1997). 

[7] Mayer, R. Using Illustrations to Promote Constructivist Learning from Science Text. Psychology of 
Science Text Comprehension. (Submitted). 

[8] Moreno, R., & Mayer, R. E. Cognitive principles of multimedia learning: The role of modality and 
contiguity. Journal of Educational Psychology, 91 , 358-368, (1999). 




144 



U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTTCE 

REPRODUCTION BASTS 



This document is covered by a signed “Reproduction Release 
(Blanket) form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a “Specific Document” Release form. 



This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release form 
(either “Specific Document” or “Blanket”). 




